returning results from python script to variable in Jupyter notebook

Question

I have a python script that returns a pandas dataframe and I want to run the script in a Jupyter notebook and then save the results to a variable.

The data are in a file called data.csv and a shortened version of the dataframe.py file whose results I want to access in my Jupyter notebook is:

# dataframe.py
import pandas as pd
import sys

def return_dataframe(file):
    df = pd.read_csv(file)
    return df

if __name__ == '__main__':
    return_dataframe(sys.argv[1])

I tried running:

data = !python dataframe.py data.csv

in my Jupyter notebook but data does not contain the dataframe that dataframe.py is supposed to return.

mechanical_meat · Accepted Answer · 2020-04-29 03:26:46Z

1

This is how I did it:

# dataframe.py 
import pandas as pd
import sys

def return_dataframe(f): # don't shadow built-in `file`
    df = pd.read_csv(f)
    return df

if __name__ == '__main__':
    return_dataframe(sys.argv[1]).to_csv(sys.stdout,index=False)

Then in the notebook you need to convert an 'IPython.utils.text.SList' into a DataFrame as shown in the comments to this question: Convert SList to Dataframe:

data = !python3 dataframe.py data.csv
df = pd.DataFrame(data=data)[0].str.split(',',expand=True)

If the DataFrame is already going to be put into CSV format then you could simply do this in the notebook:

df = pd.read_csv('data.csv')

edited Apr 29, 2020 at 3:26

answered Apr 29, 2020 at 2:45

mechanical_meat

170k25 gold badges237 silver badges231 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

tshwizz Over a year ago

The real script is a rather long data wrangling and cleaning process, I just left in the bottom line which is returning the dataframe

mechanical_meat Over a year ago

Makes sense. I hope the answer helps your process.

tshwizz Over a year ago

Any advice on how to keep the original dtypes of the different columns

mechanical_meat Over a year ago

Instead of saving and reading from CSV you can use pickle: stackoverflow.com/a/51177054/42346 which will preserve dtypes.

Collectives™ on Stack Overflow

returning results from python script to variable in Jupyter notebook

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest