5

I know this is basic but I am really struggling on how to get started and am so so so frustrated.

I have developed the basis of my app on my windows pc using juypter notebooks and now want to start working in a cloud environment. I have no clue where to start and have looked at so much but the basics don't seem to be covered.

The data to run this on is here - named JJO_Xsens_210513.csv

A very basic version of what I am trying to do is within this snippet:

import pandas as pd
import numpy as np

df = pd.read_csv("JJO_Xsens_210513.csv", skiprows=1)
df["Keep"] = np.where(df.Acc_X <= -50,1,0)

df_to_write = df[df.Keep ==1]

A long long way down the road, there should be an upload functionality and then the preprocessing code is run and the data is then summarised in many different ways to give insights which are delivered by charts and apps. I know I can (in theory) build cloud functions, and look at persistent data storage, and building a web app / web site for all of this.

I have looked at the tutorials on GCP for App Engine, looked at tutorials for python and even the basics. I have completed the Developing Apps with GCP specialisation course via Coursera.

None of this tells me how to start in a basic way of running a python script and saving some output somewhere. It all throws me in at the deepend of building apps. I just want to test that I can run my code and models in GCP.

I apologise for the whiny and emotional nature of this post, but I am at my wits end. I am trying so hard to get this running and am so ******** lost.

Can someone please help / point me in the right direction.
Thank you,
J

Improve this question
5
  • 1
    I'm not sure to understand your inquiry but, have you tried installing Gcloud SDK? this will allow you to run python targeting google objects from your local machine: cloud.google.com/sdk/docs/install, then there's a specific snipet of how to upload a CSV to GCP BigCloud using python, but not sure if that's your scenario: cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv Commented May 17, 2021 at 21:19
  • 1
    My general inquiry is how to start basic. Not running an app, but just processing data. I want to run some code on some data and save it somewhere. I will look at the links you've shared. Thanks. Commented May 18, 2021 at 8:36
  • If you are willing to do some system setup you may want to try f1 micro instance of their VPS offerings. It should give you access to a system prompt where you can setup your python with some very basic debian/ubuntu commands and get going from there. It's probably not any sort of production grade setup but will enable you to work your way up into more specialized setups from there. This may help explain the details:hbmy289.medium.com/… Commented May 18, 2021 at 11:33
  • @FloWil so your article talks me through the setup of an instance. But how do I load and run the script, and load the data to this instance? The persistent disk mentioned I believe is associated with the instance and so would be deleted once the instance is shut down - as you say that's a consideration for next steps. Commented May 18, 2021 at 11:50
  • Well if you have ssh access you should be able to also copy files up there through some form of SCP/SFTP which can use the ssh tunnel as transfer. WinSCP or some form rsync if you work on a linux desktop usually do the trick. Commented May 18, 2021 at 12:30

2 Answers 2

Reset to default
2

I you need to "just" your code run notebooks in GCP there is a service for that: AI Platform Notebooks.

From your notebook instance you can work and download data you need, just be sure your instance has the resources (memory and CPU) you'll need, and do not forget to shut the instance down when you're not using it to avoid charges.

If a notebook is too much for you, you can always run everything in a compute instance (that's a virtual machine). Just launch it from the web and ssh to it with the web application (no need to install ssh, etc.), then you can upload the code into the instance. Just remenber to use a small instance one to start ;-)

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

AI Platform looks promising. I'll look into this. Re the VM, I've done tutorials on setting up a VM and ssh'ing into it, however what is missing is how I work with it - ie load the code and run it on some data that I've loaded and writing an output... those are the steps I cannot seem to find anywhere. Could you advise on those steps?
This is the easiest way to do it: cloud.google.com/compute/docs/instances/… (just ssh from the web interface, click on the sprocket icon, and upload files from your computer).
Went with the VM option and used the upload and this code to get myself up and running sudo apt-get update sudo apt-get install git -y sudo apt-get install python3-setuptools python3-dev build-essential python3-venv -y curl bootstrap.pypa.io/get-pip.py -o get-pip.py sudo python3 get-pip.py python3 --version pip3 --version python3 -m venv env source env/bin/activate pip install -r requirements.txt Python3 filename.py
0

So I followed Inigo's advice, plus a little code I found elsewhere (starting with python on google plus a quick lab)

Create the VM as per here SSH in Run this code

sudo apt-get update
sudo apt-get install git -y
sudo apt-get install python3-setuptools python3-dev build-essential python3-venv -y

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
# wget as suggested in the docs didn't work for me
sudo python3 get-pip.py
python3 --version
pip3 --version

python3 -m venv env
source env/bin/activate

Sprocket on the right to upload the files

pip install -r requirements.txt
python3 filename.py

Thanks for the comments and tips gang. This has got me moving

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.