GCP - Insert/Load Data to BigQuery from REST API using Google Cloud Function with Python

Question

Is it possible to directly insert/stream data from a rest api with json output to bigquery in gcp using cloud function?

Doing some research but it always ask to be put in GCS bucket then create the cloud function to load the data to bigquery.

thanks in advance.

JC98 · Accepted Answer · 2020-06-24 22:40:05Z

1

Yes you can load it with no need of passing through GCS. Using this tutorial as a guide (which uses JSON files) brings us to this Source Code (used by the Cloud Function in the tutorial), as you can see all it does is load the bucket file into a blob and then to a json :

row = json.loads(blob.download_as_string())

After that is just a matter of using the insert_rows_json method :

BQ.insert_rows_json(table, json_rows=[row],
                             row_ids=[file_name],
                             retry=retry.Retry(deadline=30))

So, you can just send from your API the requests with a POST to a HTTP Cloud Function and that function use the above code to load the json fo BigQuery.

answered Jun 24, 2020 at 22:40

JC98

6345 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sandeep Mohanty Over a year ago

Hi @JC98 , can you please share the whole code..

Collectives™ on Stack Overflow

GCP - Insert/Load Data to BigQuery from REST API using Google Cloud Function with Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related