Joining pandas dataframes from array / list

Question

I am trying to merge (concat) dataframes selected from csv files. I have previously achieve this manually but want to use a (tkinter) filedialog so that the user can select one or more files to use in the dataframe. What I have is:

Read the filenames into an array until the user presses cancel - all good

while True:
    root = Tk()
    root.withdraw()
    filename = filedialog.askopenfilename(title='Open data file', filetypes=(("Comma separated values", "*.csv"),))
    if filename =="":
        break
    filenames.append(filename)

Use the filenames to generate a dataframe - all good individually (I think, as they print OK)

def read_Yokogawa(filename):
    frame = pd.read_csv(filename, header = 46, skiprows = [47, 48], low_memory = False, parse_dates = True, infer_datetime_format = True, na_values = ['+OVER', '-OVER'])
    print(frame)
    return frame

dfs=[]  #Set an empty array of dataframes
for filename in filenames:
    dfs.append([read_Yokogawa(filename)])   #Append each dataframe to the array

Concat the array of dataframes together - not good

df = pd.concat(dfs)

I get the error: TypeError: cannot concatenate object of type ''; only Series and DataFrame objs are valid

All the files have the same columns as they are generated by the same software. I have also tried pd.concat([dfs]) but get the same error.

It might help to include sample data and type of, for example, dfs[0] and dfs[1], or [i for i in dfs if not isinstance(i, pd.DataFrame)] — G. Anderson
– G. Anderson, Commented Feb 26, 2020 at 20:50
Change dfs.append([read_Yokogawa(filename)]) to dfs.append(read_Yokogawa(filename)). — Quang Hoang
– Quang Hoang, Commented Feb 26, 2020 at 20:51
I think your problem come from the square brackets. Try: dfs.append(read_Yokogawa(filename)) — Renaud
– Renaud, Commented Feb 26, 2020 at 20:54

ChemEnger · Accepted Answer · 2020-02-27 09:10:32Z

1

Both @Quang and @Renaud suggested the same thing, which turned out to be a good answer to the question.

dfs=[]
for filename in filenames:
    dfs.append(read_Yokogawa(filename))
df = pd.concat(dfs)

answered Feb 27, 2020 at 9:10

ChemEnger

1491 silver badge13 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Joining pandas dataframes from array / list

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related