BigQuery GCP Python Integration

Question

I am trying to write all my scripts in Python instead of BigQuery. I set my active project using 'glcoud config set project' but I still get this ERROR 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/analytics-supplychain-thd/jobs: Caller does not have required permission to use project analytics-supplychain-thd. Grant the caller the Owner or Editor role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=analytics-supplychain-thd and then retry (propagation of new permission may take a few minutes).

How do I fix this?

Can you please add info regarding exactly what are you trying to do? how are you trying to do it? — Alonme
– Alonme, Commented Mar 20, 2020 at 22:07
Read this link and then edit your question to be suitable for Stack Overflow. stackoverflow.com/help/minimal-reproducible-example — John Hanley
– John Hanley, Commented Mar 21, 2020 at 17:41

Federico Taranto · Accepted Answer · 2020-03-23 15:37:06Z

1

I suspect you are picking up the wrong "key".json, at least in terms of permissions for one of the operation you are trying to perform. The key currently defined [1] in GOOGLE_APPLICATION_CREDENTIALS seems not have right permission. A list of roles you should grant to the Service Account can be find here [2], anyway from your error you would need at least a primitive role as Owner or Editor. The latter depends on your needs and targets (operation you perform through such script).

You should pick up the right role for your operation and associating it to the Service Account you want to use, defining therefore an identity for it through the IAM portal UI, also doable also through the CLI or API calls.

After that be sure the client you are using is logged in with the correct service account (correct json key path).

Particularly, I used the code you gave me to test and I have been able to load the data:

import pandas_gbq 
import google.oauth2.service_account as service_account
# TODO: Set project_id to your Google Cloud Platform project ID 
project_id = "xxx-xxxx-xxxxx" 
sql = """SELECT * FROM xxx-xxxx-xxxxx.fourth_dataset.2test LIMIT 100""" 
credentials = service_account.Credentials.from_service_account_file('/home/myself/key.json')
df = pandas_gbq.read_gbq(sql, project_id=project_id, dialect="standard", credentials=credentials)

This Hope this helps!!

[1] https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable

[2] https://cloud.google.com/iam/docs/understanding-roles#primitive_roles

edited Mar 23, 2020 at 15:37

answered Mar 23, 2020 at 9:35

Federico Taranto

1325 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Krithika Srinivasan Over a year ago

This is what I tried running: import pandas_gbq # TODO: Set project_id to your Google Cloud Platform project ID project_id = "analytics-supplychain-thd" sql = """ SELECT * FROM KXS26XY.OSC_FY2022_MIXTIER_08SDCs_SCENARIO_V1 """ df = pandas_gbq.read_gbq(sql, project_id=project_id)

Federico Taranto Over a year ago

I modified my answer with your code used for troubleshooting. Please note that you should also have access at dataset level to be able to read from it.

Collectives™ on Stack Overflow

BigQuery GCP Python Integration

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related