Python Pandas Dynamically Create a Dataframe

Question

The code below will generate the desired output in ONE dataframe, however, I would like to dynamically create data frames in a FOR loop then assign the shifted value to that data frame. Example, data frame df_lag_12 would only contain column1_t12 and column2_12. Any ideas would be greatly appreciated. I attempted to dynamically create 12 dataframes using the EXEC statement, google searching seems to state this is poor practice.

import pandas as pd
list1=list(range(0,20))
list2=list(range(19,-1,-1))
d={'column1':list(range(0,20)),
   'column2':list(range(19,-1,-1))}
df=pd.DataFrame(d)
df_lags=pd.DataFrame()
for col in df.columns:
    for i in range(12,0,-1):
        df_lags[col+'_t'+str(i)]=df[col].shift(i)
    df_lags[col]=df[col].values  
print(df_lags)
for df in (range(12,0,-1)):
    exec('model_data_lag_'+str(df)+'=pd.DataFrame()')

Desired output for dymanically created dataframe DF_LAGS_12:

var_list=['column1_t12','column2_t12']
df_lags_12=df_lags[var_list]  
print(df_lags_12)

jezrael · Accepted Answer · 2017-11-04 11:25:31Z

13

I think the best is create dictionary of DataFrames:

d = {}
for i in range(12,0,-1):
    d['t' + str(i)] = df.shift(i).add_suffix('_t' + str(i))

If need specify columns first:

d = {}
cols = ['column1','column2']
for i in range(12,0,-1):
    d['t' + str(i)] = df[cols].shift(i).add_suffix('_t' + str(i))

dict comprehension solution:

d = {'t' + str(i): df.shift(i).add_suffix('_t' + str(i)) for i in range(12,0,-1)}

print (d['t10'])
    column1_t10  column2_t10
0           NaN          NaN
1           NaN          NaN
2           NaN          NaN
3           NaN          NaN
4           NaN          NaN
5           NaN          NaN
6           NaN          NaN
7           NaN          NaN
8           NaN          NaN
9           NaN          NaN
10          0.0         19.0
11          1.0         18.0
12          2.0         17.0
13          3.0         16.0
14          4.0         15.0
15          5.0         14.0
16          6.0         13.0
17          7.0         12.0
18          8.0         11.0
19          9.0         10.0

EDIT: Is it possible by globals, but much better is dictionary:

d = {}
cols = ['column1','column2']
for i in range(12,0,-1):
    globals()['df' + str(i)] =  df[cols].shift(i).add_suffix('_t' + str(i))

print (df10)
    column1_t10  column2_t10
0           NaN          NaN
1           NaN          NaN
2           NaN          NaN
3           NaN          NaN
4           NaN          NaN
5           NaN          NaN
6           NaN          NaN
7           NaN          NaN
8           NaN          NaN
9           NaN          NaN
10          0.0         19.0
11          1.0         18.0
12          2.0         17.0
13          3.0         16.0
14          4.0         15.0
15          5.0         14.0
16          6.0         13.0
17          7.0         12.0
18          8.0         11.0
19          9.0         10.0

edited Nov 4, 2017 at 11:25

answered Nov 4, 2017 at 11:10

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Kyle Over a year ago

This looks promising, would it be possible to dynamically assign these to dataframes 1-12 though?

jezrael Over a year ago

do you think df1, df10 ? It is possible, but bad practice.

Kyle Over a year ago

Okay, now I see what you are doing, I didnt realize a dictionary could hold dataframes... So, I implemented your recommended version via: d_df={} for col in model_data.drop(['usrec'],axis=1).columns: for i in range(12,0,-1): d_df['t'+str(i)]=model_data[col].shift(i).add_suffix('_t'+str(i)).to_frame() However, it will only preserve the last column of the 2300 or so columns in the original dataframe, I know of the append function, however unsure if this would work here, any ideas?

Kyle Over a year ago

Hi jezrael, do you think you could assist with this question? <stackoverflow.com/questions/47198514/…> if this is considered "taboo" sorry, I wont do that again.

jezrael Over a year ago

No, it is ideal way for ping someone ;) Give me some time, I can check it.

|

oguz ismail · Accepted Answer · 2021-06-12 09:31:13Z

2

for i in range(1, 16):
    text=f"Version{i}=pd.DataFrame()"
    exec(text)

A combination of exec and f"..." will help you do that. If you need iterating or Versions of same variable above statement will help

edited Jun 12, 2021 at 9:31

oguz ismail

51.8k16 gold badges64 silver badges83 bronze badges

answered Jun 12, 2021 at 9:25

velusamy subbarayalu

211 bronze badge

Collectives™ on Stack Overflow

Python Pandas Dynamically Create a Dataframe

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related