Powerful Google Cloud
tools for your hack
Wesley Chun
Developer Advocate, Google
Adjunct CS Faculty, Foothill College
G Suite Dev Show
goo.gl/JpBQ40
About the speaker
Developer Advocate, Google Cloud
● Mission: enable current and future
developers everywhere to be
successful using Google Cloud and
other Google developer tools & APIs
● Videos: host of the G Suite Dev Show
on YouTube
● Blogs: developers.googleblog.com &
gsuite-developers.googleblog.com
● Twitters: @wescpy, @GoogleDevs,
@GSuiteDevs
Previous experience / background
● Software engineer & architect for 20+ years
● One of the original Yahoo!Mail engineers
● Author of bestselling "Core Python" books
(corepython.com)
● Technical trainer, teacher, instructor since
1983 (Computer Science, C, Linux, Python)
● Fellow of the Python Software Foundation
● AB (Math/CS) & CMP (Music/Piano), UC
Berkeley and MSCS, UC Santa Barbara
● Adjunct Computer Science Faculty, Foothill
College (Silicon Valley)
Why and Agenda
● How Google Cloud can help with your hack
● Can't hurt to learn a bit more about cloud computing
● Cloud skills are in-demand in today's workforce
● Show how to get started with Google Cloud
Cloud computing
& Google Cloud
1
Hosting your
code
2
Google APIs
primer
3
Machine
Learning
4 5
Other APIs to
consider
6
Wrap-up
01
Cloud computing
& Google Cloud
Intro and overview
What is Google Cloud Platform?
Getting things done using someone else’s computers, especially
where someone else worries about maintenance, provisioning, system
administration, security, networking, failure recover, etc.
Google Compute Engine, Google Cloud Storage
AWS EC2 & S3; Rackspace; Joyent
Cloud service levels/"pillars"
SaaS
Software as a Service
PaaS
Platform as a Service
IaaS
Infrastructure as a Service
Google BigQuery, Cloud SQL,
Cloud Datastore, NL, Vision, Pub/Sub
AWS Kinesis, RDS; Windows Azure SQL, Docker
Google Apps Script, App Maker
Salesforce1/force.com
G Suite (Google Apps)
Yahoo!Mail, Hotmail, Salesforce, Netsuite
Google App Engine, Cloud Functions
Heroku, Cloud Foundry, Engine Yard, AWS Lambda
Summary of responsibility
SaaS
Software as a Service
Applications
Data
Runtime
Middleware
OS
Virtualization
Servers
Storage
Networking
Applications
Data
Runtime
Middleware
OS
Virtualization
Servers
Storage
Networking
IaaS
Infrastructure as a Service
Applications
Data
Runtime
Middleware
OS
Virtualization
Servers
Storage
Networking
PaaS
Platform as a Service
Managed by YOU Managed by cloud vendor
Applications
Data
Runtime
Middleware
OS
Virtualization
Servers
Storage
Networking
on-prem
all you, no cloud
Theme - Spec/Reqs
Logistics - Design app
Space - Provision HW
Food - Build & serve app
Manage - Manage app
on-prem
(DIY)
IaaS
(Compute Engine)
PaaS
(App Engine/GCF)
SaaS
(G Suite)
Pick theme
Plan party
Find space
Cook
On-call
Pick theme
Plan party
Rent hall
Cook
On-call
Pick theme
Plan party
Rent hall
Hire Caterer
Hire manager
Pick theme
Hire planner
Rent hall
Hire caterer
Hire manager
Imagine you’re hosting a party...
YOUR
NEXT
APP? google.com/datacenter
What is Google Cloud Platform?
GCP lets you build & host code (web apps, mobile
backends, web services, containers), store &
analyze data, and much more, all on Google’s
highly-scalable & reliable computing infrastructure
What is Google Cloud Platform?some Google Cloud Platform products
Compute Big Data
BigQuery
Cloud
Dataflow
Cloud
Dataproc
Cloud
Datalab
Cloud
Pub/Sub
Genomics
Storage & Databases
Cloud
Storage
Cloud
Bigtable
Cloud
Datastore
Cloud SQL
Cloud
Spanner
Persistent
Disk
Machine Learning
Cloud ML
Engine
Cloud
Vision API
Cloud
Speech APIs
Cloud Natural
Language API
Cloud
Translation
API
Cloud
Jobs API
Data
Studio
Cloud
Dataprep
Cloud Video
Intelligence
API
AutoML suite
Compute
Engine
App
Engine
Kubernete
s Engine
GPU
Cloud
Functions
Container-
Optimized OS
Identity & Security
Cloud IAM
Cloud Resource
Manager
Cloud Security
Scanner
Key
Management
Service
BeyondCorp
Data Loss
Prevention API
Identity-Aware
Proxy
Security Key
Enforcement
Internet of Things
Cloud IoT
Core
Transfer
Appliance
Cloud
Firestore
02
Hosting your
code
on GCP serverless/PaaS
What is Google Cloud Platform?some Google Cloud Platform products (tl;dr)
Compute Big Data
BigQuery
Cloud
Dataflow
Cloud
Dataproc
Cloud
Datalab
Cloud
Pub/Sub
Genomics
Storage & Databases
Cloud
Storage
Cloud
Bigtable
Cloud
Datastore
Cloud SQL
Cloud
Spanner
Persistent
Disk
Machine Learning
Cloud ML
Engine
Cloud
Vision API
Cloud
Speech APIs
Cloud Natural
Language API
Cloud
Translation
API
Cloud
Jobs API
Data
Studio
Cloud
Dataprep
Cloud Video
Intelligence
API
AutoML auite
Compute
Engine
App
Engine
Kubernete
s Engine
GPU
Cloud
Functions
Container-
Optimized OS
Identity & Security
Cloud IAM
Cloud Resource
Manager
Cloud Security
Scanner
Key
Management
Service
BeyondCorp
Data Loss
Prevention API
Identity-Aware
Proxy
Security Key
Enforcement
Transfer
Appliance
Cloud
Firestore
Internet of Things
Cloud IoT
Core
>
Google Compute Engine configurable
VMs of all shapes & sizes, from
"micro" to 416 vCPUs, 11.75 TB RAM,
64 TB HDD/SSD plus Google Cloud
Storage for blobs/cloud data lake
(Debian, CentOS, CoreOS, SUSE, Red Hat Enterprise Linux,
Ubuntu, FreeBSD; Windows Server 2008 R2, 2012 R2, 2016)
cloud.google.com/compute
cloud.google.com/storage
Yeah, we got VMs & big disk… but why?
Serverless: what & why
● What is serverless?
○ Misnomer
○ "No worries"
○ Developers focus on writing code & solving business problems*
● Why serverless?
○ Fastest growing segment of cloud... per analyst research*:
■ $1.9B (2016) and $4.25B (2018) ⇒ $7.7B (2021) and $14.93B (2023)
○ What if you go viral? Autoscaling: your new best friend
○ What if you don't? Code not running? You're not paying.
* in USD; source:Forbes (May 2018), MarketsandMarkets™ & CB Insights (Aug 2018)
Google Compute Engine, Google Cloud Storage
AWS EC2 & S3; Rackspace; Joyent
SaaS
Software as a Service
PaaS
Platform as a Service
IaaS
Infrastructure as a Service
G Suite (Google Apps)
Yahoo!Mail, Hotmail, Salesforce, Netsuite
Google App Engine, Cloud Functions
Heroku, Cloud Foundry, Engine Yard, AWS Lambda
Google BigQuery, Cloud SQL,
Cloud Datastore, NL, Vision, Pub/Sub
AWS Kinesis, RDS; Windows Azure SQL, Docker
Serverless: PaaS-y compute/processing
Google Apps Script, App Maker
Salesforce1/force.com
Running Code: App Engine
Got a great app idea? Now what?
VMs? Operating systems? Big disk?
Web servers? Load balancing?
Database servers? Autoscaling?
With Google App Engine, you don't
think about those. Just upload
your code; we do everything else.
>
cloud.google.com/appengine
Why does App Engine exist?
● Focus on app not DevOps
○ Web app
○ Mobile backend
○ Cloud service
● Enhance productivity
● Deploy globally
● Fully-managed
● Auto-scaling
● Pay-per-use
● Familiar languages
App Engine to the rescue!!
● Focus on app not DevOps
● Enhance productivity
● Deploy globally
● Fully-managed
● Auto-scaling
● Pay-per-use
● Familiar standard runtimes
● 2nd gen std platforms
○ Python 3.7
○ Java 8, 11
○ PHP 7.2
○ Go 1.11
○ JS/Node.js 8, 10
○ Ruby 2.5
Hello World (3 files: Python "MVP")
app.yaml
runtime: python37
main.py
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello():
return 'Hello World!'
requirements.txt
Flask==1.0.2
Deploy:
$ gcloud app deploy
Access globally:
https://PROJECT_ID.appspot.com
Open source repo at
github.com/GoogleCloudPlatform/python-docs-samples/
tree/master/appengine/standard_python37/hello_world
App Engine demo
Python Flask QuickStart tutorial
(also Java, Node.js, PHP, Go, Ruby)
Running Code: Cloud Functions
Don't have an entire app? Just want
to deploy small microservices or
"RPCs" online globally? That's what
Google Cloud Functions are for!
(+Firebase version for mobile apps)
cloud.google.com/functions
firebase.google.com/products/functions
Why does Cloud Functions exist?
● Don't have entire app?
○ No framework "overhead" (LAMP, MEAN...)
○ Deploy microservices
● Event-driven
○ Triggered via HTTP or background events
■ Pub/Sub, Cloud Storage, Firebase, etc.
○ Auto-scaling & highly-available; pay per use
● Flexible development environment
○ Cmd-line or developer console
● Cloud Functions for Firebase
○ Mobile app use-cases
● Available runtimes
○ JS/Node.js 6, 8, 10
○ Python 3.7
○ Go 1.11, 1.12
○ Java 8
main.py
def hello_world(request):
return 'Hello World!'
Deploy:
$ gcloud functions deploy hello --runtime python37 --trigger-http
Access globally (curl):
curl -X POST https://GCP_REGION-PROJECT_ID.cloudfunctions.net/hello 
-H "Content-Type:application/json"
Access globally (browser):
GCP_REGION-PROJECT_ID.cloudfunctions.net/hello
Hello World (Python "MVP")
No cmd-line
access?
Use in-browser
dev environment!
● setup
● code
● deploy
● test
Cloud Functions demo
Python GCF QuickStart tutorial
(also in Node.js & Go)
Running Code: Cloud Run
Got a containerized app? Want its
flexibility along with the convenience
of serverless that's fully-managed
plus auto-scales? Google Cloud Run is
exactly what you're looking for!
cloud.google.com/run
Code, build, deploy
.js .rb .go
.sh.py ...
● Any language, library, binary
○ HTTP port, stateless
● Bundle into container
○ Build w/Docker OR
○ Google Cloud Build
○ Image ⇒ Container Registry
● Deploy to Cloud Run (managed or GKE)
StateHTTP
https://yourservice.run.app
03
Google APIs
primer
How to access Cloud APIs
The first word on Security
Authentication ("authn") vs authorization ("authz")
● authn: you are who you say you are
○ login & passwd
○ handprint authentication
○ retina scan
● authz: okay, you are who you say you are, but can you haz data?
○ OAuth2 - mostly authz, but some authn
○ Mostly about 3rd-party access to data
○ Users must give YOUR code access to THEIR data
○ Most of the time when you see "auth", it refers to authz
● Some refer to this as "consent" vs. "credentials…" which is which?
OAuth2 or
API key
HTTP-based REST APIs 1
HTTP
2
Google APIs request-response workflow
● Application makes request
● Request received by service
● Process data, return response
● Results sent to application
(typical client-server model)
Cloud/GCP console
console.cloud.google.com
● Hub of all developer activity
● Applications == projects
○ New project for new apps
○ Projects have a billing acct
● Manage billing accounts
○ Financial instrument required
○ Personal or corporate credit cards,
Free Trial, and education grants
● Access GCP product settings
● Manage users & security
● Manage APIs in devconsole
Navigating the Cloud console
Collaborating & sharing
● Sharing: great way to see student work or have teams work on one project
● IAM (Identity & Access Mgmt): owner & editor most useful access levels
cloud.google.com/iam/docs/understanding-roles
● View application statistics
● En-/disable Google APIs
● Obtain application credentials
Using Google APIs
goo.gl/RbyTFD
API manager aka Developers Console (devconsole)
console.developers.google.com
04
Machine
Learning
Access AI/ML w/API calls
Puppy
or
Muffin?
Machine learning is learning
from rules plus experience.
Rules
Data
Traditional
Programming
Answers
Answers
Data
RulesMachine
Learning
Training Phase
Answers
Data
Rules/modelMachine
Learning
Model
Data Predictions
Inference Phase
Google Translate
Google Photos
Did you ever stop
to notice this app
has a search bar?!?
GCP Machine Learning APIs
● Gain insights from data using GCP's
pre-trained machine learning models
● Leverage the same technology as
Google Translate, Photos, and Assistant
● Requires ZERO prior knowledge of ML
● If you can call an API, you can use AI/ML!
Vision Video
Intelligence
Speech
(S2T & T2S)
Natural
Language
Translation
Machine Learning: Cloud Vision
Google Cloud Vision API
enables developers to extract
metadata & understand the
content of an image
cloud.google.com/vision
labeling = VISION.images().annotate(body=body).execute().get('responses')
for labels in labeling:
if 'labelAnnotations' in labels:
print('** Labels detected (and confidence score):')
for label in labels['labelAnnotations']:
print(('%.2f%%' % (
label['score']*100.)).ljust(10), label['description'])
if 'faceAnnotations' in labels:
print('n** Facial features detected (and likelihood):')
for label, value in labels['faceAnnotations'][0].items():
if label.endswith('Likelihood'):
print(label.split('Likelihood')[0].ljust(16),
value.lower().replace('_', ' '))
Vision: image analysis & metadata extraction
$ python viz_demo.py
** Labels detected (and confidence score):
89.94% Sitting
86.09% Interior design
82.08% Furniture
81.52% Table
80.85% Room
79.04% White-collar worker
76.19% Office
68.18% Conversation
60.96% Window
60.07% Desk
** Facial features detected (and likelihood):
anger very unlikely
joy very likely
underExposed very unlikely
sorrow very unlikely
surprise very unlikely
headwear very unlikely
blurred very unlikely
Vision: image analysis & metadata extraction
Cloud Vision demo
quickdraw.withgoogle.com
Machine Learning: Cloud Natural Language
Google Cloud Natural Language API
reveals the structure & meaning
of text; also performs content
classification and sentiment
analysis; multi-lingual
cloud.google.com/language
Simple sentiment & classification analysis
TEXT = '''Google, headquartered in Mountain View, unveiled the new
Android phone at the Consumer Electronics Show. Sundar Pichai said
in his keynote that users love their new Android phones.'''
print('TEXT:', TEXT)
data = {'type': 'PLAIN_TEXT', 'content': TEXT}
NL = discovery.build('language', 'v1', developerKey=API_KEY)
# sentiment analysis
sent = NL.documents().analyzeSentiment(
body={'document': data}).execute().get('documentSentiment')
print('nSENTIMENT: score (%s), magnitude (%s)' % (sent['score'], sent['magnitude']))
# content classification
print('nCATEGORIES:')
cats = NL.documents().classifyText(body={'document': data}).execute().get('categories')
for cat in cats:
print('* %s (%s)' % (cat['name'][1:], cat['confidence']))
Simple sentiment & classification analysis
$ python nl_sent_simple.py
TEXT: Google, headquartered in Mountain View, unveiled the new Android
phone at the Consumer Electronics Show. Sundar Pichai said in
his keynote that users love their new Android phones.
SENTIMENT: score (0.3), magnitude (0.6)
CATEGORIES:
* Internet & Telecom (0.76)
* Computers & Electronics (0.64)
* News (0.56)
Machine Learning: Cloud Speech
Google Cloud Speech APIs enable
developers to convert
speech-to-text and vice versa
cloud.google.com/speech
cloud.google.com/text-to-speech
Text-to-Speech: synthsizing audio text
# request body (with text body using 16-bit linear PCM audio encoding)
body = {
'input': {'text': text},
'voice': {
'languageCode': 'en-US',
'ssmlGender': 'FEMALE',
},
'audioConfig': {'audioEncoding': 'LINEAR16'},
}
# call Text-to-Speech API to synthesize text (write to text.wav file)
T2S = discovery.build('texttospeech', 'v1', developerKey=API_KEY)
audio = T2S.text().synthesize(body=body).execute().get('audioContent')
with open('text.wav', 'wb') as f:
f.write(base64.b64decode(audio))
Speech-to-Text: transcribing audio text
# request body (16-bit linear PCM audio content, i.e., from text.wav)
body = {
'audio': {'content': audio},
'config': {
'languageCode': 'en-US',
'encoding': 'LINEAR16',
},
}
# call Speech-to-Text API to recognize text
S2T = discovery.build('speech', 'v1', developerKey=API_KEY)
rsp = S2T.speech().recognize(
body=body).execute().get('results')[0]['alternatives'][0]
print('** %.2f%% confident of this transcript:n%r' % (
rsp['confidence']*100., rsp['transcript']))
Speech-to-Text: transcribing audio text
$ python s2t_demo.py
** 92.03% confident of this transcript:
'Google headquarters in Mountain View unveiled the new
Android phone at the Consumer Electronics Show Sundar
pichai said in his keynote that users love their new
Android phones'
Machine Learning: Cloud Video Intelligence
Google Cloud Video Intelligence
API makes videos searchable, and
discoverable, by extracting
metadata. Other features: object
tracking, shot change detection,
and text detection
cloud.google.com/video-intelligence
Video intelligence: make videos searchable
# request body (single payload, base64 binary video)
body = {
"inputContent": video,
"features": ['LABEL_DETECTION', 'SPEECH_TRANSCRIPTION'],
"videoContext": {"speechTranscriptionConfig": {"languageCode": 'en-US'}},
}
# perform video shot analysis followed by speech analysis
VINTEL = discovery.build('videointelligence', 'v1', developerKey=API_KEY)
resource = VINTEL.videos().annotate(body=body).execute().get('name')
while True:
results = VINTEL.operations().get(name=resource).execute()
if results.get('done'):
break
time.sleep(random.randrange(8)) # expo-backoff probably better
Video intelligence: make videos searchable
# display shot labels followed by speech transcription
for labels in results['response']['annotationResults']:
if 'shotLabelAnnotations' in labels:
print('n** Video shot analysis labeling')
for shot in labels['shotLabelAnnotations']:
seg = shot['segments'][0]
print(' - %s (%.2f%%)' % (
shot['entity']['description'], seg['confidence']*100.))
if 'speechTranscriptions' in labels:
print('** Speech transcription')
speech = labels['speechTranscriptions'][0]['alternatives'][0]
print(' - %r (%.2f%%)' % (
speech['transcript'], speech['confidence']*100.))
Video intelligence: make videos searchable
$ python3 vid_demo.py you-need-a-hug.mp4
** Video shot analysis labeling
- vacation (30.62%)
- fun (61.53%)
- interaction (38.93%)
- summer (57.10%)
** Speech transcription
- 'you need a hug come here' (79.27%)
Machine Learning: Cloud Translation
Access Google Translate
programmatically through this
API; translate an arbitrary
string into any supported
language using state-of-the-art
Neural Machine Translation
cloud.google.com/translate
Translate some texts: “Hello World!”
const {Translate} = require('@google-cloud/translate');
const translate = new Translate({projectConfig});
const text = 'Hello World!'; // Text to translate
const target = 'ru'; // Target language
// Translates some text into Russian
const translation = await translate.translate(text, {from: 'en', to:
target}));
// Translation: Привет, мир!
console.log('Translation:', translation[0]);
Translate Methods
const translate = new Translate({projectConfig});
// Translates text to target language
const translation = await translate.translate(text, target);
// Gives the list of supported languages with abbrevations
const languages = await translate.getLanguages();
// Detects the given language
const language = await translate.detect('сайн байна уу');
Pro Tip:
● If translating many strings, it's more performant to combine
together into 1 massive string to send a single request rather
than making 100s of small requests.
● Concatenate strings with delimiter 'n', then split once you
get the response. (Obviously use a different delimiter if your
strings have NEWLINEs in them.)
Combining Translations
// Bunch of texts needs to be translated
const texts = ['Morning World!', 'Evening World!', 'Evening World!'];
const target = 'ru';
// Translates some text into Russian
const [translation] = await translate.translate(texts.join('n'),
target);
console.log(translation.split('n'));
Output: [ 'Утренний мир!', 'Вечерний Мир!', 'Ночной Мир!' ]
Machine Learning: AutoML
AutoML: a suite of cloud APIs for
developers with limited machine
learning expertise; chooses the best
models & allows for further training
of those models for your data
(Translation, Vision, Natural Language,
Video Intelligence, Tables)
cloud.google.com/automl
cloud.google.com/automl-tables
● What is it, and how does it work?
○ Google Cloud ML APIs use pre-trained models
○ Perhaps those models less suitable for your data
○ Further customize/train our models for your data
○ Without requiring sophisticated ML knowledge
○ Translate, Vision, Natural Language, Video Intelligence, Tables
○ cloud.google.com/automl
● Steps
a. Prep your training data
b. Create dataset
c. Import items into dataset
d. Create/train model
e. Evaluate/validate model
f. Make predictions
Cloud AutoML
Full Spectrum of AI & ML Offerings
App Developer Data Scientist ML Scientist/Researcher
Use pre-built models Use/extend OSS SDK
ML EngineAuto ML
Build custom models
ML APIs
05
Other APIs to
consider
What else may be useful?
Storing Data: Cloud SQL
SQL servers in the cloud
High-performance, fully-managed
600MB to 416GB RAM; up to 64 vCPUs
Up to 10 TB storage; 40,000 IOPS
Types:
MySQL
Postgres
SQLServer (2019)
cloud.google.com/sql
Storing Data: Cloud Datastore
Cloud Datastore: a fully-
managed, highly-scalable
NoSQL database for your web
and mobile applications
cloud.google.com/datastore
Storing Data: Firebase
Firebase data is stored
as JSON & synchronized in
real-time to every
connected client; other
tools + FB == v2 mobile
development platform
firebase.google.com
Storing Data: Cloud Firestore
The best of both worlds: the
next generation of Cloud
Datastore (w/product rebrand)
plus features from the
Firebase realtime database
cloud.google.com/firestore
● Ordinary database - explicitiy query database for
new updates (but when?)
● Realtime database - automatically receive deltas
when database updated (setup client listener object)
● Highly scalable database
● Multi-regional data replication
● Strong consistency (vs. eventual consistency): read
your writes!
Cloud Firestore key features
Cloud Firestore data model
collections subcollections
Cloud Firestore easy querying
Cloud Firestore
demo
bit.ly/firestore-game
Storing and Analyzing Data: BigQuery
Google BigQuery: a fast, highly
scalable, cost-effective, and
fully-managed data warehouse in the
cloud for analytics with built-in
machine learning; issue SQL queries
across multi-terabytes of data
cloud.google.com/bigquery
BigQuery: querying Shakespeare words
TITLE = "The top 10 most common words in all of Shakespeare's works"
QUERY = '''
SELECT LOWER(word) AS word, sum(word_count) AS count
FROM [bigquery-public-data:samples.shakespeare]
GROUP BY word ORDER BY count DESC LIMIT 10
'''
rsp = BQ.query(body={'query': QUERY}, projectId=PROJ_ID).execute()
print('n*** Results for %r:n' % TITLE)
for col in rsp['schema']['fields']: # HEADERS
print(col['name'].upper(), end='t')
print()
for row in rsp['rows']: # DATA
for col in row['f']:
print(col['v'], end='t')
print()
Top 10 most common Shakespeare words
$ python bq_shake.py
*** Results for "The most common words in all of Shakespeare's works":
WORD COUNT
the 29801
and 27529
i 21029
to 20957
of 18514
a 15370
you 14010
my 12936
in 11722
that 11519
G Suite: Google Drive
Drive API allows developers to read,
write, control permissions/sharing,
import/export files, and more!
developers.google.com/drive
List (first 100) files/folders in Google Drive
from __future__ import print_function
from googleapiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools
SCOPES = 'https://www.googleapis.com/auth/drive.metadata.readonly'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))
files = DRIVE.files().list().execute().get('files', [])
for f in files:
print(f['name'], f['mimeType'])
Listing your files
goo.gl/ZIgf8k
Automate photo albums
OR
G Suite: Google Sheets
Sheets API gives you programmatic
access to spreadsheets; perform
(w/code) almost any action you can
do from the web interface as a user
developers.google.com/sheets
Try our Node.js customized reporting tool codelab:
g.co/codelabs/sheets
Why use the Sheets API?
data visualization
customized reports
Sheets as a data source
Migrate SQL data to a Sheet
# read SQL data then create new spreadsheet & add rows into it
FIELDS = ('ID', 'Customer Name', 'Product Code',
'Units Ordered', 'Unit Price', 'Status')
cxn = sqlite3.connect('db.sqlite')
cur = cxn.cursor()
rows = cur.execute('SELECT * FROM orders').fetchall()
cxn.close()
rows.insert(0, FIELDS)
DATA = {'properties': {'title': 'Customer orders'}}
SHEET_ID = SHEETS.spreadsheets().create(body=DATA,
fields='spreadsheetId').execute().get('spreadsheetId')
SHEETS.spreadsheets().values().update(spreadsheetId=SHEET_ID, range='A1',
body={'values': rows}, valueInputOption='RAW').execute()
Migrate SQL data
to Sheets
goo.gl/N1RPwC
G Suite: Google Docs & Slides
Docs & Slides APIs give you access
to read or write documents and
presentations programmatically so
you can autogenerate them with data
integrated from various sources
developers.google.com/docs
developers.google.com/slides
Try our Node.js BigQuery GitHub license analyzer codelab:
g.co/codelabs/slides
Why use the Slides API?
data visualization
presentable reports
Try our Node.js Markdown-to-Google-Slides generator:
github.com/gsuitedevs/md2googleslides
Why use the Slides API?
customized presentations
Replace text & images from template deck
requests = [
# (global) search-and-replace text
{'replaceAllText': {
'findText': '{{TITLE}}',
'replaceText': 'Hello World!',
}},
# replace text-based image placeholders (global)
{'replaceAllShapesWithImage': {
'imageUrl': IMG_URL, # link to product logo
'replaceMethod': 'CENTER_INSIDE',
'containsText': {'text': '{{LOGO}}'},
}},
]
SLIDES.presentations().batchUpdate(body={'requests': requests},
presentationId=DECK_ID, fields='').execute()
Replacing text
and images
goo.gl/o6EFwk
Other Google APIs & platforms
● Firebase (mobile development platform + RT DB)
○ firebase.google.com
● Google Data Studio (data visualization, dashboards, etc.)
○ marketingplatform.google.com/about/data-studio
● Actions on Google/Assistant/DialogFlow (voice apps)
○ developers.google.com/actions
● YouTube (Data, Analytics, and Livestreaming APIs)
○ developers.google.com/youtube
● Google Maps (Maps, Routes, and Places APIs)
○ developers.google.com/maps
● Flutter (native apps [Android, iOS, web] w/1 code base[!])
○ flutter.dev
06
Wrap-up
Summary & resources for
hackers
● Key GCP product code samples from this talk
github.com/GoogleCloudPlatform/hackathon-toolkit
● GCP documentation - cloud.google.com/{appengine,functions,vision,
language,speech,text-to-speech,translate,automl,firestore,bigquery}
● Like GCP? Wanna use it in class or your research lab? Send your profs
to cloud.google.com/edu to apply for teaching or research credits!
● Other docs
○ Firebase - firebase.google.com
○ G Suite - developers.google.com/{gsuite,drive,docs,sheets,slides}
○ Free trial (ignore) and Always Free (FYI) - cloud.google.com/free
Resources
Best use of GCP/
Cloud challenge
Use any GCP API to qualify; Google Cloud
affiliated APIs (G Suite, Firebase, Maps) allowed!
Every member of the winning team gets:
● Google Home Mini
● Patagonia Backpack
● Cloud Pillow
● Acrylic Trophy
● Water Bottle
$100 in GCP credits
Check for an email from Major League
Hacking (MLH) to activate your coupon
for free $100 worth of GCP credits!
(NOTE: must exceed Always Free tier to
incur billing, so you may not use much)
WARNING: avoid the GCP $300 free trial!! 💳 😓
Questions/Help?
Wesley Chun
@wescpy
Progress bars: goo.gl/69EJVw

Powerful Google Cloud tools for your hack

  • 1.
    Powerful Google Cloud toolsfor your hack Wesley Chun Developer Advocate, Google Adjunct CS Faculty, Foothill College G Suite Dev Show goo.gl/JpBQ40 About the speaker Developer Advocate, Google Cloud ● Mission: enable current and future developers everywhere to be successful using Google Cloud and other Google developer tools & APIs ● Videos: host of the G Suite Dev Show on YouTube ● Blogs: developers.googleblog.com & gsuite-developers.googleblog.com ● Twitters: @wescpy, @GoogleDevs, @GSuiteDevs Previous experience / background ● Software engineer & architect for 20+ years ● One of the original Yahoo!Mail engineers ● Author of bestselling "Core Python" books (corepython.com) ● Technical trainer, teacher, instructor since 1983 (Computer Science, C, Linux, Python) ● Fellow of the Python Software Foundation ● AB (Math/CS) & CMP (Music/Piano), UC Berkeley and MSCS, UC Santa Barbara ● Adjunct Computer Science Faculty, Foothill College (Silicon Valley)
  • 2.
    Why and Agenda ●How Google Cloud can help with your hack ● Can't hurt to learn a bit more about cloud computing ● Cloud skills are in-demand in today's workforce ● Show how to get started with Google Cloud Cloud computing & Google Cloud 1 Hosting your code 2 Google APIs primer 3 Machine Learning 4 5 Other APIs to consider 6 Wrap-up 01 Cloud computing & Google Cloud Intro and overview
  • 3.
    What is GoogleCloud Platform? Getting things done using someone else’s computers, especially where someone else worries about maintenance, provisioning, system administration, security, networking, failure recover, etc. Google Compute Engine, Google Cloud Storage AWS EC2 & S3; Rackspace; Joyent Cloud service levels/"pillars" SaaS Software as a Service PaaS Platform as a Service IaaS Infrastructure as a Service Google BigQuery, Cloud SQL, Cloud Datastore, NL, Vision, Pub/Sub AWS Kinesis, RDS; Windows Azure SQL, Docker Google Apps Script, App Maker Salesforce1/force.com G Suite (Google Apps) Yahoo!Mail, Hotmail, Salesforce, Netsuite Google App Engine, Cloud Functions Heroku, Cloud Foundry, Engine Yard, AWS Lambda
  • 4.
    Summary of responsibility SaaS Softwareas a Service Applications Data Runtime Middleware OS Virtualization Servers Storage Networking Applications Data Runtime Middleware OS Virtualization Servers Storage Networking IaaS Infrastructure as a Service Applications Data Runtime Middleware OS Virtualization Servers Storage Networking PaaS Platform as a Service Managed by YOU Managed by cloud vendor Applications Data Runtime Middleware OS Virtualization Servers Storage Networking on-prem all you, no cloud Theme - Spec/Reqs Logistics - Design app Space - Provision HW Food - Build & serve app Manage - Manage app on-prem (DIY) IaaS (Compute Engine) PaaS (App Engine/GCF) SaaS (G Suite) Pick theme Plan party Find space Cook On-call Pick theme Plan party Rent hall Cook On-call Pick theme Plan party Rent hall Hire Caterer Hire manager Pick theme Hire planner Rent hall Hire caterer Hire manager Imagine you’re hosting a party...
  • 5.
  • 6.
    What is GoogleCloud Platform? GCP lets you build & host code (web apps, mobile backends, web services, containers), store & analyze data, and much more, all on Google’s highly-scalable & reliable computing infrastructure What is Google Cloud Platform?some Google Cloud Platform products Compute Big Data BigQuery Cloud Dataflow Cloud Dataproc Cloud Datalab Cloud Pub/Sub Genomics Storage & Databases Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Cloud Spanner Persistent Disk Machine Learning Cloud ML Engine Cloud Vision API Cloud Speech APIs Cloud Natural Language API Cloud Translation API Cloud Jobs API Data Studio Cloud Dataprep Cloud Video Intelligence API AutoML suite Compute Engine App Engine Kubernete s Engine GPU Cloud Functions Container- Optimized OS Identity & Security Cloud IAM Cloud Resource Manager Cloud Security Scanner Key Management Service BeyondCorp Data Loss Prevention API Identity-Aware Proxy Security Key Enforcement Internet of Things Cloud IoT Core Transfer Appliance Cloud Firestore
  • 7.
    02 Hosting your code on GCPserverless/PaaS What is Google Cloud Platform?some Google Cloud Platform products (tl;dr) Compute Big Data BigQuery Cloud Dataflow Cloud Dataproc Cloud Datalab Cloud Pub/Sub Genomics Storage & Databases Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Cloud Spanner Persistent Disk Machine Learning Cloud ML Engine Cloud Vision API Cloud Speech APIs Cloud Natural Language API Cloud Translation API Cloud Jobs API Data Studio Cloud Dataprep Cloud Video Intelligence API AutoML auite Compute Engine App Engine Kubernete s Engine GPU Cloud Functions Container- Optimized OS Identity & Security Cloud IAM Cloud Resource Manager Cloud Security Scanner Key Management Service BeyondCorp Data Loss Prevention API Identity-Aware Proxy Security Key Enforcement Transfer Appliance Cloud Firestore Internet of Things Cloud IoT Core
  • 8.
    > Google Compute Engineconfigurable VMs of all shapes & sizes, from "micro" to 416 vCPUs, 11.75 TB RAM, 64 TB HDD/SSD plus Google Cloud Storage for blobs/cloud data lake (Debian, CentOS, CoreOS, SUSE, Red Hat Enterprise Linux, Ubuntu, FreeBSD; Windows Server 2008 R2, 2012 R2, 2016) cloud.google.com/compute cloud.google.com/storage Yeah, we got VMs & big disk… but why? Serverless: what & why ● What is serverless? ○ Misnomer ○ "No worries" ○ Developers focus on writing code & solving business problems* ● Why serverless? ○ Fastest growing segment of cloud... per analyst research*: ■ $1.9B (2016) and $4.25B (2018) ⇒ $7.7B (2021) and $14.93B (2023) ○ What if you go viral? Autoscaling: your new best friend ○ What if you don't? Code not running? You're not paying. * in USD; source:Forbes (May 2018), MarketsandMarkets™ & CB Insights (Aug 2018)
  • 9.
    Google Compute Engine,Google Cloud Storage AWS EC2 & S3; Rackspace; Joyent SaaS Software as a Service PaaS Platform as a Service IaaS Infrastructure as a Service G Suite (Google Apps) Yahoo!Mail, Hotmail, Salesforce, Netsuite Google App Engine, Cloud Functions Heroku, Cloud Foundry, Engine Yard, AWS Lambda Google BigQuery, Cloud SQL, Cloud Datastore, NL, Vision, Pub/Sub AWS Kinesis, RDS; Windows Azure SQL, Docker Serverless: PaaS-y compute/processing Google Apps Script, App Maker Salesforce1/force.com Running Code: App Engine Got a great app idea? Now what? VMs? Operating systems? Big disk? Web servers? Load balancing? Database servers? Autoscaling? With Google App Engine, you don't think about those. Just upload your code; we do everything else. > cloud.google.com/appengine
  • 10.
    Why does AppEngine exist? ● Focus on app not DevOps ○ Web app ○ Mobile backend ○ Cloud service ● Enhance productivity ● Deploy globally ● Fully-managed ● Auto-scaling ● Pay-per-use ● Familiar languages App Engine to the rescue!! ● Focus on app not DevOps ● Enhance productivity ● Deploy globally ● Fully-managed ● Auto-scaling ● Pay-per-use ● Familiar standard runtimes ● 2nd gen std platforms ○ Python 3.7 ○ Java 8, 11 ○ PHP 7.2 ○ Go 1.11 ○ JS/Node.js 8, 10 ○ Ruby 2.5
  • 11.
    Hello World (3files: Python "MVP") app.yaml runtime: python37 main.py from flask import Flask app = Flask(__name__) @app.route('/') def hello(): return 'Hello World!' requirements.txt Flask==1.0.2 Deploy: $ gcloud app deploy Access globally: https://PROJECT_ID.appspot.com Open source repo at github.com/GoogleCloudPlatform/python-docs-samples/ tree/master/appengine/standard_python37/hello_world App Engine demo Python Flask QuickStart tutorial (also Java, Node.js, PHP, Go, Ruby)
  • 12.
    Running Code: CloudFunctions Don't have an entire app? Just want to deploy small microservices or "RPCs" online globally? That's what Google Cloud Functions are for! (+Firebase version for mobile apps) cloud.google.com/functions firebase.google.com/products/functions Why does Cloud Functions exist? ● Don't have entire app? ○ No framework "overhead" (LAMP, MEAN...) ○ Deploy microservices ● Event-driven ○ Triggered via HTTP or background events ■ Pub/Sub, Cloud Storage, Firebase, etc. ○ Auto-scaling & highly-available; pay per use ● Flexible development environment ○ Cmd-line or developer console ● Cloud Functions for Firebase ○ Mobile app use-cases ● Available runtimes ○ JS/Node.js 6, 8, 10 ○ Python 3.7 ○ Go 1.11, 1.12 ○ Java 8
  • 13.
    main.py def hello_world(request): return 'HelloWorld!' Deploy: $ gcloud functions deploy hello --runtime python37 --trigger-http Access globally (curl): curl -X POST https://GCP_REGION-PROJECT_ID.cloudfunctions.net/hello -H "Content-Type:application/json" Access globally (browser): GCP_REGION-PROJECT_ID.cloudfunctions.net/hello Hello World (Python "MVP") No cmd-line access? Use in-browser dev environment! ● setup ● code ● deploy ● test
  • 14.
    Cloud Functions demo PythonGCF QuickStart tutorial (also in Node.js & Go) Running Code: Cloud Run Got a containerized app? Want its flexibility along with the convenience of serverless that's fully-managed plus auto-scales? Google Cloud Run is exactly what you're looking for! cloud.google.com/run
  • 15.
    Code, build, deploy .js.rb .go .sh.py ... ● Any language, library, binary ○ HTTP port, stateless ● Bundle into container ○ Build w/Docker OR ○ Google Cloud Build ○ Image ⇒ Container Registry ● Deploy to Cloud Run (managed or GKE) StateHTTP https://yourservice.run.app 03 Google APIs primer How to access Cloud APIs
  • 16.
    The first wordon Security Authentication ("authn") vs authorization ("authz") ● authn: you are who you say you are ○ login & passwd ○ handprint authentication ○ retina scan ● authz: okay, you are who you say you are, but can you haz data? ○ OAuth2 - mostly authz, but some authn ○ Mostly about 3rd-party access to data ○ Users must give YOUR code access to THEIR data ○ Most of the time when you see "auth", it refers to authz ● Some refer to this as "consent" vs. "credentials…" which is which? OAuth2 or API key HTTP-based REST APIs 1 HTTP 2 Google APIs request-response workflow ● Application makes request ● Request received by service ● Process data, return response ● Results sent to application (typical client-server model)
  • 17.
    Cloud/GCP console console.cloud.google.com ● Hubof all developer activity ● Applications == projects ○ New project for new apps ○ Projects have a billing acct ● Manage billing accounts ○ Financial instrument required ○ Personal or corporate credit cards, Free Trial, and education grants ● Access GCP product settings ● Manage users & security ● Manage APIs in devconsole Navigating the Cloud console
  • 18.
    Collaborating & sharing ●Sharing: great way to see student work or have teams work on one project ● IAM (Identity & Access Mgmt): owner & editor most useful access levels cloud.google.com/iam/docs/understanding-roles ● View application statistics ● En-/disable Google APIs ● Obtain application credentials Using Google APIs goo.gl/RbyTFD API manager aka Developers Console (devconsole) console.developers.google.com
  • 19.
  • 20.
    Puppy or Muffin? Machine learning islearning from rules plus experience.
  • 21.
  • 23.
  • 24.
    Google Translate Google Photos Didyou ever stop to notice this app has a search bar?!?
  • 25.
    GCP Machine LearningAPIs ● Gain insights from data using GCP's pre-trained machine learning models ● Leverage the same technology as Google Translate, Photos, and Assistant ● Requires ZERO prior knowledge of ML ● If you can call an API, you can use AI/ML! Vision Video Intelligence Speech (S2T & T2S) Natural Language Translation Machine Learning: Cloud Vision Google Cloud Vision API enables developers to extract metadata & understand the content of an image cloud.google.com/vision
  • 26.
    labeling = VISION.images().annotate(body=body).execute().get('responses') forlabels in labeling: if 'labelAnnotations' in labels: print('** Labels detected (and confidence score):') for label in labels['labelAnnotations']: print(('%.2f%%' % ( label['score']*100.)).ljust(10), label['description']) if 'faceAnnotations' in labels: print('n** Facial features detected (and likelihood):') for label, value in labels['faceAnnotations'][0].items(): if label.endswith('Likelihood'): print(label.split('Likelihood')[0].ljust(16), value.lower().replace('_', ' ')) Vision: image analysis & metadata extraction $ python viz_demo.py ** Labels detected (and confidence score): 89.94% Sitting 86.09% Interior design 82.08% Furniture 81.52% Table 80.85% Room 79.04% White-collar worker 76.19% Office 68.18% Conversation 60.96% Window 60.07% Desk ** Facial features detected (and likelihood): anger very unlikely joy very likely underExposed very unlikely sorrow very unlikely surprise very unlikely headwear very unlikely blurred very unlikely Vision: image analysis & metadata extraction
  • 27.
    Cloud Vision demo quickdraw.withgoogle.com MachineLearning: Cloud Natural Language Google Cloud Natural Language API reveals the structure & meaning of text; also performs content classification and sentiment analysis; multi-lingual cloud.google.com/language
  • 28.
    Simple sentiment &classification analysis TEXT = '''Google, headquartered in Mountain View, unveiled the new Android phone at the Consumer Electronics Show. Sundar Pichai said in his keynote that users love their new Android phones.''' print('TEXT:', TEXT) data = {'type': 'PLAIN_TEXT', 'content': TEXT} NL = discovery.build('language', 'v1', developerKey=API_KEY) # sentiment analysis sent = NL.documents().analyzeSentiment( body={'document': data}).execute().get('documentSentiment') print('nSENTIMENT: score (%s), magnitude (%s)' % (sent['score'], sent['magnitude'])) # content classification print('nCATEGORIES:') cats = NL.documents().classifyText(body={'document': data}).execute().get('categories') for cat in cats: print('* %s (%s)' % (cat['name'][1:], cat['confidence'])) Simple sentiment & classification analysis $ python nl_sent_simple.py TEXT: Google, headquartered in Mountain View, unveiled the new Android phone at the Consumer Electronics Show. Sundar Pichai said in his keynote that users love their new Android phones. SENTIMENT: score (0.3), magnitude (0.6) CATEGORIES: * Internet & Telecom (0.76) * Computers & Electronics (0.64) * News (0.56)
  • 29.
    Machine Learning: CloudSpeech Google Cloud Speech APIs enable developers to convert speech-to-text and vice versa cloud.google.com/speech cloud.google.com/text-to-speech Text-to-Speech: synthsizing audio text # request body (with text body using 16-bit linear PCM audio encoding) body = { 'input': {'text': text}, 'voice': { 'languageCode': 'en-US', 'ssmlGender': 'FEMALE', }, 'audioConfig': {'audioEncoding': 'LINEAR16'}, } # call Text-to-Speech API to synthesize text (write to text.wav file) T2S = discovery.build('texttospeech', 'v1', developerKey=API_KEY) audio = T2S.text().synthesize(body=body).execute().get('audioContent') with open('text.wav', 'wb') as f: f.write(base64.b64decode(audio))
  • 30.
    Speech-to-Text: transcribing audiotext # request body (16-bit linear PCM audio content, i.e., from text.wav) body = { 'audio': {'content': audio}, 'config': { 'languageCode': 'en-US', 'encoding': 'LINEAR16', }, } # call Speech-to-Text API to recognize text S2T = discovery.build('speech', 'v1', developerKey=API_KEY) rsp = S2T.speech().recognize( body=body).execute().get('results')[0]['alternatives'][0] print('** %.2f%% confident of this transcript:n%r' % ( rsp['confidence']*100., rsp['transcript'])) Speech-to-Text: transcribing audio text $ python s2t_demo.py ** 92.03% confident of this transcript: 'Google headquarters in Mountain View unveiled the new Android phone at the Consumer Electronics Show Sundar pichai said in his keynote that users love their new Android phones'
  • 31.
    Machine Learning: CloudVideo Intelligence Google Cloud Video Intelligence API makes videos searchable, and discoverable, by extracting metadata. Other features: object tracking, shot change detection, and text detection cloud.google.com/video-intelligence Video intelligence: make videos searchable # request body (single payload, base64 binary video) body = { "inputContent": video, "features": ['LABEL_DETECTION', 'SPEECH_TRANSCRIPTION'], "videoContext": {"speechTranscriptionConfig": {"languageCode": 'en-US'}}, } # perform video shot analysis followed by speech analysis VINTEL = discovery.build('videointelligence', 'v1', developerKey=API_KEY) resource = VINTEL.videos().annotate(body=body).execute().get('name') while True: results = VINTEL.operations().get(name=resource).execute() if results.get('done'): break time.sleep(random.randrange(8)) # expo-backoff probably better
  • 32.
    Video intelligence: makevideos searchable # display shot labels followed by speech transcription for labels in results['response']['annotationResults']: if 'shotLabelAnnotations' in labels: print('n** Video shot analysis labeling') for shot in labels['shotLabelAnnotations']: seg = shot['segments'][0] print(' - %s (%.2f%%)' % ( shot['entity']['description'], seg['confidence']*100.)) if 'speechTranscriptions' in labels: print('** Speech transcription') speech = labels['speechTranscriptions'][0]['alternatives'][0] print(' - %r (%.2f%%)' % ( speech['transcript'], speech['confidence']*100.)) Video intelligence: make videos searchable $ python3 vid_demo.py you-need-a-hug.mp4 ** Video shot analysis labeling - vacation (30.62%) - fun (61.53%) - interaction (38.93%) - summer (57.10%) ** Speech transcription - 'you need a hug come here' (79.27%)
  • 33.
    Machine Learning: CloudTranslation Access Google Translate programmatically through this API; translate an arbitrary string into any supported language using state-of-the-art Neural Machine Translation cloud.google.com/translate Translate some texts: “Hello World!” const {Translate} = require('@google-cloud/translate'); const translate = new Translate({projectConfig}); const text = 'Hello World!'; // Text to translate const target = 'ru'; // Target language // Translates some text into Russian const translation = await translate.translate(text, {from: 'en', to: target})); // Translation: Привет, мир! console.log('Translation:', translation[0]);
  • 34.
    Translate Methods const translate= new Translate({projectConfig}); // Translates text to target language const translation = await translate.translate(text, target); // Gives the list of supported languages with abbrevations const languages = await translate.getLanguages(); // Detects the given language const language = await translate.detect('сайн байна уу'); Pro Tip: ● If translating many strings, it's more performant to combine together into 1 massive string to send a single request rather than making 100s of small requests. ● Concatenate strings with delimiter 'n', then split once you get the response. (Obviously use a different delimiter if your strings have NEWLINEs in them.)
  • 35.
    Combining Translations // Bunchof texts needs to be translated const texts = ['Morning World!', 'Evening World!', 'Evening World!']; const target = 'ru'; // Translates some text into Russian const [translation] = await translate.translate(texts.join('n'), target); console.log(translation.split('n')); Output: [ 'Утренний мир!', 'Вечерний Мир!', 'Ночной Мир!' ] Machine Learning: AutoML AutoML: a suite of cloud APIs for developers with limited machine learning expertise; chooses the best models & allows for further training of those models for your data (Translation, Vision, Natural Language, Video Intelligence, Tables) cloud.google.com/automl cloud.google.com/automl-tables
  • 36.
    ● What isit, and how does it work? ○ Google Cloud ML APIs use pre-trained models ○ Perhaps those models less suitable for your data ○ Further customize/train our models for your data ○ Without requiring sophisticated ML knowledge ○ Translate, Vision, Natural Language, Video Intelligence, Tables ○ cloud.google.com/automl ● Steps a. Prep your training data b. Create dataset c. Import items into dataset d. Create/train model e. Evaluate/validate model f. Make predictions Cloud AutoML Full Spectrum of AI & ML Offerings App Developer Data Scientist ML Scientist/Researcher Use pre-built models Use/extend OSS SDK ML EngineAuto ML Build custom models ML APIs
  • 37.
    05 Other APIs to consider Whatelse may be useful? Storing Data: Cloud SQL SQL servers in the cloud High-performance, fully-managed 600MB to 416GB RAM; up to 64 vCPUs Up to 10 TB storage; 40,000 IOPS Types: MySQL Postgres SQLServer (2019) cloud.google.com/sql
  • 38.
    Storing Data: CloudDatastore Cloud Datastore: a fully- managed, highly-scalable NoSQL database for your web and mobile applications cloud.google.com/datastore Storing Data: Firebase Firebase data is stored as JSON & synchronized in real-time to every connected client; other tools + FB == v2 mobile development platform firebase.google.com
  • 39.
    Storing Data: CloudFirestore The best of both worlds: the next generation of Cloud Datastore (w/product rebrand) plus features from the Firebase realtime database cloud.google.com/firestore ● Ordinary database - explicitiy query database for new updates (but when?) ● Realtime database - automatically receive deltas when database updated (setup client listener object) ● Highly scalable database ● Multi-regional data replication ● Strong consistency (vs. eventual consistency): read your writes! Cloud Firestore key features
  • 40.
    Cloud Firestore datamodel collections subcollections Cloud Firestore easy querying
  • 41.
    Cloud Firestore demo bit.ly/firestore-game Storing andAnalyzing Data: BigQuery Google BigQuery: a fast, highly scalable, cost-effective, and fully-managed data warehouse in the cloud for analytics with built-in machine learning; issue SQL queries across multi-terabytes of data cloud.google.com/bigquery
  • 42.
    BigQuery: querying Shakespearewords TITLE = "The top 10 most common words in all of Shakespeare's works" QUERY = ''' SELECT LOWER(word) AS word, sum(word_count) AS count FROM [bigquery-public-data:samples.shakespeare] GROUP BY word ORDER BY count DESC LIMIT 10 ''' rsp = BQ.query(body={'query': QUERY}, projectId=PROJ_ID).execute() print('n*** Results for %r:n' % TITLE) for col in rsp['schema']['fields']: # HEADERS print(col['name'].upper(), end='t') print() for row in rsp['rows']: # DATA for col in row['f']: print(col['v'], end='t') print() Top 10 most common Shakespeare words $ python bq_shake.py *** Results for "The most common words in all of Shakespeare's works": WORD COUNT the 29801 and 27529 i 21029 to 20957 of 18514 a 15370 you 14010 my 12936 in 11722 that 11519
  • 43.
    G Suite: GoogleDrive Drive API allows developers to read, write, control permissions/sharing, import/export files, and more! developers.google.com/drive List (first 100) files/folders in Google Drive from __future__ import print_function from googleapiclient import discovery from httplib2 import Http from oauth2client import file, client, tools SCOPES = 'https://www.googleapis.com/auth/drive.metadata.readonly' store = file.Storage('storage.json') creds = store.get() if not creds or creds.invalid: flow = client.flow_from_clientsecrets('client_secret.json', SCOPES) creds = tools.run_flow(flow, store) DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http())) files = DRIVE.files().list().execute().get('files', []) for f in files: print(f['name'], f['mimeType']) Listing your files goo.gl/ZIgf8k
  • 44.
    Automate photo albums OR GSuite: Google Sheets Sheets API gives you programmatic access to spreadsheets; perform (w/code) almost any action you can do from the web interface as a user developers.google.com/sheets
  • 45.
    Try our Node.jscustomized reporting tool codelab: g.co/codelabs/sheets Why use the Sheets API? data visualization customized reports Sheets as a data source Migrate SQL data to a Sheet # read SQL data then create new spreadsheet & add rows into it FIELDS = ('ID', 'Customer Name', 'Product Code', 'Units Ordered', 'Unit Price', 'Status') cxn = sqlite3.connect('db.sqlite') cur = cxn.cursor() rows = cur.execute('SELECT * FROM orders').fetchall() cxn.close() rows.insert(0, FIELDS) DATA = {'properties': {'title': 'Customer orders'}} SHEET_ID = SHEETS.spreadsheets().create(body=DATA, fields='spreadsheetId').execute().get('spreadsheetId') SHEETS.spreadsheets().values().update(spreadsheetId=SHEET_ID, range='A1', body={'values': rows}, valueInputOption='RAW').execute() Migrate SQL data to Sheets goo.gl/N1RPwC
  • 46.
    G Suite: GoogleDocs & Slides Docs & Slides APIs give you access to read or write documents and presentations programmatically so you can autogenerate them with data integrated from various sources developers.google.com/docs developers.google.com/slides Try our Node.js BigQuery GitHub license analyzer codelab: g.co/codelabs/slides Why use the Slides API? data visualization presentable reports
  • 47.
    Try our Node.jsMarkdown-to-Google-Slides generator: github.com/gsuitedevs/md2googleslides Why use the Slides API? customized presentations Replace text & images from template deck requests = [ # (global) search-and-replace text {'replaceAllText': { 'findText': '{{TITLE}}', 'replaceText': 'Hello World!', }}, # replace text-based image placeholders (global) {'replaceAllShapesWithImage': { 'imageUrl': IMG_URL, # link to product logo 'replaceMethod': 'CENTER_INSIDE', 'containsText': {'text': '{{LOGO}}'}, }}, ] SLIDES.presentations().batchUpdate(body={'requests': requests}, presentationId=DECK_ID, fields='').execute() Replacing text and images goo.gl/o6EFwk
  • 48.
    Other Google APIs& platforms ● Firebase (mobile development platform + RT DB) ○ firebase.google.com ● Google Data Studio (data visualization, dashboards, etc.) ○ marketingplatform.google.com/about/data-studio ● Actions on Google/Assistant/DialogFlow (voice apps) ○ developers.google.com/actions ● YouTube (Data, Analytics, and Livestreaming APIs) ○ developers.google.com/youtube ● Google Maps (Maps, Routes, and Places APIs) ○ developers.google.com/maps ● Flutter (native apps [Android, iOS, web] w/1 code base[!]) ○ flutter.dev 06 Wrap-up Summary & resources for hackers
  • 49.
    ● Key GCPproduct code samples from this talk github.com/GoogleCloudPlatform/hackathon-toolkit ● GCP documentation - cloud.google.com/{appengine,functions,vision, language,speech,text-to-speech,translate,automl,firestore,bigquery} ● Like GCP? Wanna use it in class or your research lab? Send your profs to cloud.google.com/edu to apply for teaching or research credits! ● Other docs ○ Firebase - firebase.google.com ○ G Suite - developers.google.com/{gsuite,drive,docs,sheets,slides} ○ Free trial (ignore) and Always Free (FYI) - cloud.google.com/free Resources Best use of GCP/ Cloud challenge Use any GCP API to qualify; Google Cloud affiliated APIs (G Suite, Firebase, Maps) allowed! Every member of the winning team gets: ● Google Home Mini ● Patagonia Backpack ● Cloud Pillow ● Acrylic Trophy ● Water Bottle
  • 50.
    $100 in GCPcredits Check for an email from Major League Hacking (MLH) to activate your coupon for free $100 worth of GCP credits! (NOTE: must exceed Always Free tier to incur billing, so you may not use much) WARNING: avoid the GCP $300 free trial!! 💳 😓 Questions/Help? Wesley Chun @wescpy Progress bars: goo.gl/69EJVw