Keep pandas structure with numpy/scikit functions

Question

I'm using the excellent read_csv()function from pandas, which gives:

In [31]: data = pandas.read_csv("lala.csv", delimiter=",")

In [32]: data
Out[32]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 12083 entries, 0 to 12082
Columns: 569 entries, REGIONC to SCALEKER
dtypes: float64(51), int64(518)

but when i apply a function from scikit-learn i loose the informations about columns:

from sklearn import preprocessing
preprocessing.scale(data)

gives numpy array.

Is there a way to apply scikit or numpy function to DataFrames without loosing the information?

Ahmed Fasih · Accepted Answer · 2017-02-02 02:37:22Z

19

This can be done by wrapping the returned data in a dataframe, with index and columns information in.

import pandas as pd
pd.DataFrame(preprocessing.scale(data), index = data.index, columns = data.columns)

edited Feb 2, 2017 at 2:37

Ahmed Fasih

7,0378 gold badges62 silver badges103 bronze badges

answered Feb 11, 2013 at 14:37

Mermoz

15.6k17 gold badges62 silver badges85 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mbatchkarov · Accepted Answer · 2017-01-12 18:12:10Z

A (slightly naive) way would be to store the structure of your data frame, i.e. its columns and index, separately, and then create a new data frame from your preprocessed results like so:

In [15]: data = np.zeros((2,2))

In [16]: data
Out[16]: 
array([[ 0.,  0.],
       [ 0.,  0.]])

In [17]: from pandas import DataFrame

In [21]: df  = DataFrame(data, index = ['first', 'second'], columns=['c1','c2'])

In [22]: df
Out[22]: 
        c1  c2
first    0   0
second   0   0

In [26]: i = df.index

In [27]: c = df.columns

# generate new data as a numpy array    
In [29]: df  = DataFrame(np.random.rand(2,2), index=i, columns=c)

In [30]: df
Out[30]: 
              c1        c2
first   0.821354  0.936703
second  0.138376  0.482180

As you can see in Out[22], we start off with a data frame, and then in In[29] we place some new data inside the frame, leaving the rows and columns unchanged. I am assuming your preprocessing will not shuffle the rows/ columns of the data.

Collectives™ on Stack Overflow

Keep pandas structure with numpy/scikit functions

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related