putting numpy ndarray data into pandas

Question

I want to put the following data into pandas for further analysis.

import numpy as np
import pandas as pd
from pandas import DataFrame

data = np.array([[[1, 1, 1, np.nan, 1], [np.nan, 1, 1, 1, 1]],
                 [[2, np.nan, 2, 2, 2], [2, np.nan, 2, 2, 2]],
                 [[3, 3, 3, np.nan, 3], [3, 3, 3, 3, np.nan]]])

pnda = pd.Series(data)

print pnda

But the following error occurs:

Exception: Data must be 1-dimensional

What is the good way of doing it? My further analysis is to filling the np.nan values by interpolation with cubic or polynomial method and output the result as numpy array.

@Ffisegydd it seems that DataFrame only accepts 2-D arrays... — Saullo G. P. Castro
– Saullo G. P. Castro, Commented May 2, 2014 at 16:24
Ah yes sorry. I think DataFrames are the way to go rather than Series (which is typically a 1D "array" I think) but I don't know how to handle ND arrays greater than N. — Ffisegydd
– Ffisegydd, Commented May 2, 2014 at 16:26
@SaulloCastro yes, because in pandas there is builtin function for interpolating the missing data — user3235542
– user3235542, Commented May 2, 2014 at 16:26

Paul H · Accepted Answer · 2014-05-02 17:25:07Z

3

Try using a panel:

import numpy as np
import pandas as pd

data = np.array([[[1, 1, 1, np.nan, 1], [np.nan, 1, 1, 1, 1]],
                 [[2, np.nan, 2, 2, 2], [2, np.nan, 2, 2, 2]],
                 [[3, 3, 3, np.nan, 3], [3, 3, 3, 3, np.nan]]])

x = pd.Panel(data)
x

<class 'pandas.core.panel.Panel'>
Dimensions: 3 (items) x 2 (major_axis) x 5 (minor_axis)
Items axis: 0 to 2
Major_axis axis: 0 to 1
Minor_axis axis: 0 to 4

And...

print(x.loc[0])
    0  1  2   3  4
0   1  1  1 NaN  1
1 NaN  1  1   1  1

answered May 2, 2014 at 17:25

Paul H

68.7k23 gold badges165 silver badges139 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

TomAugspurger Over a year ago

Not that interpolate isn't implemented on Panels iirc, which was the OP's original need. You'll probably need to iterate over the items of the panel.

Paul H Over a year ago

@TomAugspurger or maybe a nested apply 0_o

Saullo G. P. Castro · Accepted Answer · 2014-05-02 17:51:02Z

2

Based on your comments you can achieve what you want if you reshape data, interpolate using the DataFrame.interpolate() method and then return the array to its original value. It works for pandas 0.13.1.

df = pd.DataFrame(data.reshape(2, -1))
df.interpolate(axis=1).values.reshape(data.shape)
#array([[[1, 1, 1, 1, 1],
#        [1, 1, 1, 1, 1]],
#
#       [[2, 2, 2, 2, 2],
#        [2, 2, 2, 2, 2]],
#
#       [[3, 3, 3, 3, 3],
#        [3, 3, 3, 3, 3]]], dtype=int64)

answered May 2, 2014 at 17:51

Saullo G. P. Castro

59.4k28 gold badges191 silver badges244 bronze badges

Collectives™ on Stack Overflow

putting numpy ndarray data into pandas

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related