BigQuery Python API issue

Question

I spend all day today trying to set up python BigQuery API for one of my new projects. I went through the whole process of

Creating a New Project
Enabling billing for the project
Enabling the BigQuery API
Creating a service account to connect to BigQuery API

I try the simplest example

import os
from google.cloud import bigquery

def main():
    # [START bigquery_quickstart]
    # Imports the Google Cloud client library

    # Instantiates a client
    creds = "absolute-path-to-file.json"
    os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = creds
    print(creds)
    bigquery_client = bigquery.Client().from_service_account_json(creds)

    # The name for the new dataset
    dataset_name = 'my_new_dataset'

    # Prepares the new dataset
    dataset = bigquery_client.dataset(dataset_name)

    # Creates the new dataset
    dataset.create()

    print('Dataset {} created.'.format(dataset.name))
# [END bigquery_quickstart]
if __name__ == '__main__':
    main()

and get the following error

Traceback (most recent call last):
  File "prepare-upload.py", line 121, in <module>
    main()
  File "prepare-upload.py", line 116, in main
    dataset.create()
  File "//anaconda/envs/tpot/lib/python3.5/site-packages/google/cloud/bigquery/dataset.py", line 431, in create
    method='POST', path=path, data=self._build_resource())
  File "//anaconda/envs/tpot/lib/python3.5/site-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
google.cloud.exceptions.Forbidden: 403 POST https://www.googleapis.com/bigquery/v2/projects/kobas-public-datasets-182301/datasets: Access Not Configured. BigQuery API has not been used in project 203593156286 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/bigquery.googleapis.com/overview?project=203593156286 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

The APIs are enabled and I am not sure what is the issue. What puzzles me is that I have done the same process for two of my other projects and it worked. One more observation. The error suggests to go to

https://console.developers.google.com/apis/api/bigquery.googleapis.com/overview?project=203593156286

however, this link is outdated and instead should be

https://console.developers.google.com/apis/api/bigquery-json.googleapis.com/overview?project=203593156286

Thanks.

Vasily Bronsky · Accepted Answer · 2017-10-10 11:47:32Z

1

Try something like this mb it will help (I'm using it for auth in pyhton 2.7):

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/path_to_key/xxx.json"

credentials = GoogleCredentials.get_application_default()

bigquery_service = build('bigquery', 'v2', credentials=credentials)

Also i don't see any definition for your_project_id in your code. For example for query i'm using:

query_response = query_request.query(
        projectId=your_project_id,
        body=query_data).execute()

answered Oct 10, 2017 at 11:47

Vasily Bronsky

4332 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Javier Montón · Accepted Answer · 2017-12-29 09:55:29Z

1

In the machine where you are executing the Python code, is the "access to Cloud APIs" enabled?

When you are creating a VM you can change it:

answered Dec 29, 2017 at 9:55

Javier Montón

6,0063 gold badges25 silver badges32 bronze badges

Comments

irvifa · Accepted Answer · 2020-04-12 14:26:24Z

0

As long as you have the GOOGLE_APPLICATION_CREDENTIALS environment variable you can initialize your client using the following code:

from google.cloud import bigquery
client = bigquery.Client()

Also make sure the service account have sufficient access to your project. So in any case, let's say you have multiple project and you want to access each of that using a service account, then either you should:

Make sure your service account have multiple access to your projects
Create different service account for each projects

edited Apr 12, 2020 at 14:26

answered Apr 12, 2020 at 7:49

irvifa

2,1132 gold badges19 silver badges21 bronze badges

Collectives™ on Stack Overflow

BigQuery Python API issue

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related