pandas read_csv path issue

Question

I can easily read a csv file if I go:

pd.read_csv(r'C:\\yourpath\yourcsvfile.csv')

But I wanna try path variables instead of hardcode my path. So I tried:

processed_data_path=os.path.join(os.path.pardir,'data','processed')
train_file_path=os.path.abspath(os.path.join(processed_data_path,'train.csv'))
test_file_path=os.path.abspath(os.path.join(processed_data_path,'test.csv'))

the print result is:

C:\yourpath\train.csv
C:\yourpath\test.csv

So it looks alright. But when I tried the code below it throws a file does not exist error:

train_df = pd.read_csv(train_file_path)

where is the problem?

forgot to mention that I'm using windows with python 3.6

Gene Xu
– Gene Xu

2018-04-01 05:03:48 +00:00
Commented Apr 1, 2018 at 5:03 — Gene Xu
– Gene Xu, Commented Apr 1, 2018 at 5:03
show the value that you are assigning to train_file_path

Vivek Kalyanarangan
– Vivek Kalyanarangan

2018-04-01 05:12:39 +00:00
Commented Apr 1, 2018 at 5:12 — Vivek Kalyanarangan
– Vivek Kalyanarangan, Commented Apr 1, 2018 at 5:12

MaxU - stand with Ukraine · Accepted Answer · 2018-04-01 10:12:02Z

2

I would use pathlib module, which is very convenient and natively supported by modern Pandas versions:

Demo:

try:
    from pathlib import Path
except ImportError:             # Python 2
    from pathlib2 import Path

In [49]: p = Path('D:\\')

use overloaded operator / for joining paths:

In [50]: f = p / 'temp' / '.data' / '1.txt'

In [51]: f
Out[51]: WindowsPath('D:/temp/.data/1.txt')

In [52]: pd.read_csv(f)
Out[52]:
        Val
0  0.120000
1  0.320000
2  0.439999
3  0.560000
4  0.599999

answered Apr 1, 2018 at 10:12

MaxU - stand with Ukraine

212k37 gold badges402 silver badges436 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

toheedNiaz · Accepted Answer · 2018-04-01 05:14:03Z

1

Maybe adding a check before calling the parser will help you to identify the issue and it is always a good pratice to check path before parsing to make sure we dont't hit the wall of exceptions :

if os.path.exists(train_file_path):
           train_df = pd.read_csv(train_file_path)
else:
           print(train_file_path)

answered Apr 1, 2018 at 5:14

toheedNiaz

1,4551 gold badge10 silver badges14 bronze badges

Comments

Jake · Accepted Answer · 2020-09-09 14:12:28Z

0

For VSCode users: The issue might be that VSCode sets current working directory to workspace root folder. My solution was:

To set current working directory to whatever file you are executing at the time:

File > Preferences > Settings > Python > Data Science > Execute in File Dir

Thanks to user brch: Python in VSCode: Set working directory to python file's path everytime

answered Sep 9, 2020 at 14:12

Jake

1111 silver badge5 bronze badges

Collectives™ on Stack Overflow

pandas read_csv path issue

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related