Parsing a CSV into a database for an API using Python?

Question

I'm gonna use data from a .csv to train a model to predict user activity on google ads (impressions, clicks) in relation to the weather for a given day. And I have a .csv that contains 6000+ recordings of this info and want to parse it into a database using Python.

I tried making a df in pandas but for some reason the whole table isn't shown. The middle columns (there's about 7 columns I think) and rows (numbered over 6000 as I mentioned) are replaced with '...' when I print the table so I'm not sure if the entirety of the information is being stored and if this will be usable.

My next attempt will possible be SQLite but since it's local memory, will this interfere with someone else making requests to my API endpoint if I don't have the db actively open at all times?

Thanks in advance.

If you decide on sqlite, you can import CSV files into tables from its command line shell program sqlite3, no code required: sqlite.org/cli.html#csv_import — Shawn
– Shawn, Commented Nov 11, 2018 at 21:56

vencaslac · Accepted Answer · 2018-11-11 22:01:30Z

1

If you used pd.read_csv() i can assure you all of the info is there, it's just not displaying it.

You can check by doing something like print(df['Column_name_you_are_interested_in'].tolist()) just to make sure though. You can also use the various count type methods in pandas to make sure all of your lines are there.

Panadas is pretty versatile so it shouldn't have trouble with 6000 lines

answered Nov 11, 2018 at 22:01

vencaslac

2,8811 gold badge21 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

HöjdeN Over a year ago

Thanks! I did use read.csv(). I would've included the code but I deleted it all last night stressing over trying to fix everything with an SQL solution of some type.

HöjdeN Over a year ago

But also if you don't mind answering, how should I convert dates used to fetch weather data into numerical values for when I want to train a model with them? I'm very green behind the ears when it comes to ML.

vencaslac Over a year ago

I would use something like datetime to convert to UNIX time and go from there. Just in case you're unfamiliar with it, UNIX time is an integer number of seconds from jan 1st 1970... it should be useful to you

Collectives™ on Stack Overflow

Parsing a CSV into a database for an API using Python?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related