List of dataframes in a loop in python

Question

I have list of data frames, a list of 279 elements and each element is made of 10 data frames. So for example List_DATA[0][9] gives me the 9th data frame of the first list. I am trying to create a different list of data "List_Selected" in a loop using the values of "List_DATA":

List_Selected=[]
for i in range(0,279):
    List_Selected_temp=[]
    for j in range(0,10):
        List_Selected_temp.append(List_DATA[i][j][(List_DATA[i][j]['A']>3) & (List_DATA[i][j]['A']<5)])
        List_Selected.append(List_Selected_temp)

The problem is in the end I get identical data-frames values for all the 279 lists. So I have a bug somewhere and I cannot find it.I guess the lists are being overwritten, can anyone maybe spot the mistake in my code?

I think (I could be wrong, of course) you want to remove one indentation of List_Selected.append(List_Selected_temp) so that it is appending in for i instead of inside for j — Karina
– Karina, Commented Sep 2, 2021 at 11:46
@Karina, Thank you for your comment, you're right. Could you please write it as an answer? — the phoenix
– the phoenix, Commented Sep 2, 2021 at 11:57

narutoArea51 · Accepted Answer · 2021-09-02 12:03:33Z

1

Try this

List_Selected=[]
for i in range(0,279):
    List_Selected_temp=[]
    for j in range(0,10):
        df = List_DATA[i][j]
        List_Selected_temp.append(df.loc[(df['A']>3) & (df['A']<5)]
    List_Selected.append(List_Selected_temp)

edited Sep 2, 2021 at 12:03

answered Sep 2, 2021 at 11:51

narutoArea51

13112 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

narutoArea51 Over a year ago

@the phoenix let me know if my answer helps you solve your problem.

the phoenix Over a year ago

@naturoArea51, thank you for your answer and help, unfortunately it did not work. The mistake was found.

narutoArea51 Over a year ago

@thephoenix that's ok.

Niv Dudovitch · Accepted Answer · 2021-09-02 12:21:20Z

0

A nice upgrade in terms of runtime that uses Python's multiprocess: Using concurrent module

import concurrent

def paralel_cond_func(curr_list_data):
    List_Selected_temp=[]
    for j in range(0,10):
        List_Selected_temp.append(curr_list_data[j][(curr_list_data[j]['A']>3) & (curr_list_data[j]['A']<5)])
    return List_Selected_temp

# number of processes
executor = concurrent.futures.ProcessPoolExecutor(10)

futures = [executor.submit(paralel_cond_func, group)
           for group in List_DATA]

concurrent.futures.wait(futures)
List_Selected = [res.result() for res in futures]

answered Sep 2, 2021 at 12:21

Niv Dudovitch

1,6789 silver badges15 bronze badges

Comments

Karina · Accepted Answer · 2021-09-02 12:27:48Z

0

List_Selected.append(...) should be outside of the for j loop, but inside of the for i loop.

List_Selected=[]
for i in range(0,279):
    List_Selected_temp=[]
    for j in range(0,10):
        List_Selected_temp.append(List_DATA[i][j][(List_DATA[i][j]['A']>3) & (List_DATA[i][j]['A']<5)])
    List_Selected.append(List_Selected_temp)

Having it inside the for j loop making it appending 279*10 times, which is most probably unwanted behavior.

answered Sep 2, 2021 at 12:27

Karina

1,3002 gold badges7 silver badges16 bronze badges

Collectives™ on Stack Overflow

List of dataframes in a loop in python

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related