How to Auth to Google Cloud using Service Account in Python?

Question

I'm trying to make a project that will upload Google Storage JSON file to BigQuery (just automate something that is done manually now), and I'd like to use a 'service account' for this as my script is going to be run on daily basis.

After reading everything I can found about using service account, I'm still struggling to authenticate.

Here is what i've done so far:

Created JSON key file for service account
Installed client libraries: pip install --upgrade google-cloud-bigquery
Installed Google Cloud SDK according to: https://cloud.google.com/sdk/docs/
Run export GOOGLE_APPLICATION_CREDENTIALS=<path_to_service_account_file> with key path specified correctly

Now I'm trying to run the following python script:

from google.cloud import bigquery
bigquery_client = bigquery.Client()

But I get this error:

google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credential and re-run the application. For more information, please see https://developers.google.com/accounts/docs/application-default-credentials.

I'm quite new to both Python and the Google Cloud API so possbily missed something. Can someone point out where/what was wrong in my steps above or point me to clear instruction for dummys about setting up and running simple script with BigQuery using service account?

Willian Fuks · Accepted Answer · 2017-06-02 15:09:15Z

85

I usually set this variable in the python script itself, something like:

import os
from google.cloud.bigquery.client import Client

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path_to_json_file'
bq_client = Client()

answered Jun 2, 2017 at 15:09

Willian Fuks

11.9k10 gold badges55 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Madhi Over a year ago

setting the credentials locally and running the script works. But when I am creating a docker image for the script it shows that JSON file is not there! How do I add service JSON file to the docker image

Willian Fuks Over a year ago

@Madhi I recommend asking this question on SO explaining what you are trying to do (I imagine this is something related to not properly adding the file in the build docker process but more information is needed in your case)

Madhi Over a year ago

Thanks for the reply comment. I figured out the solution and everything is working fine inside a docker image. Thanks!

Alberto MQ Over a year ago

@Madhi please post your solutions for others to see

Axel R. · Accepted Answer · 2022-12-16 18:33:08Z

45

If you have implemented more fine-grained control over service accounts permissions and you have an app that needs to use several of them (say one for Pub/Sub, one for storage), then you would have to set the GOOGLE_APPLICATION_CREDENTIALS environment variable before creating each client.

Instead, you can load your credentials separately and pass them to their appropriate clients like so:

import json
    
from google.cloud import storage
from google.oauth2 import service_account
    
project_id = 'my-test-project'
    
with open('/path/to/my/service_keys/storage_service.json') as source:
    info = json.load(source)

storage_credentials = service_account.Credentials.from_service_account_info(info)
    
storage_client = storage.Client(project=project_id, credentials=storage_credentials)

Just make sure in your IAM console that the account has the right permissions to perform the operations you need it to do, but luckily in that case the error messages are really informative.

edited Dec 16, 2022 at 18:33

Axel R.

1,3408 silver badges22 bronze badges

answered May 11, 2018 at 4:33

tarikki

2,3382 gold badges26 silver badges33 bronze badges

3 Comments

mowen10 Over a year ago

Hi, I know this is an old post, but I just had a quick question, as I am stuck with setting up my service account. Is it a case that once you set your key, you download it and store it where you want it? or there is a way to find the location as to where this is stored? this is where I am confused. Need to paste path of service.json in my python file that I downloaded to run a dataflow pipeline.

tarikki Over a year ago

@mowem10 You can always download the key again from GCP and then move it to where the script can find it 😊

Michał Zawadzki Mar 17 at 10:02

This doesn't work anymore - fails with AttributeError: 'Credentials' object has no attribute 'authorize'. (google-cloud-storage 2.19.0, google-auth 2.38.0, google-auth-oauthlib 1.2.1). It seems there's no longer a way to use service account credentials in the SDK, other than by using GOOGLE_APPLICATION_CREDENTIALS , which is considered bad practice by Google.

Matthias Baetens · Accepted Answer · 2017-06-02 12:32:19Z

16

Are you running the script in the same command line session as the one you set your environment variable in using export? If not, you might want to look into setting it for your user or system (see this question for more info).

Another option that might make things even easier and takes care of these things automatically is using the gcloud cli tool. If you look at the second option here under the How the Application Default Credentials work, you can find info on how to use it to manage the credentials for you (gcloud auth login and gcloud auth application-default login)

answered Jun 2, 2017 at 12:32

Matthias Baetens

1,58313 silver badges18 bronze badges

2 Comments

Marcin Wojnarski Over a year ago

In my case, the gcloud auth application-default login command was the right (and non-obvious) solution. Thanks!

minou Over a year ago

This gcloud auth application-default login requirement was so hard to find!

Manju N · Accepted Answer · 2023-08-27 07:38:54Z

6

I simply set credentials as a variable in python code, not a good idea to show credentials directly in python code though :)

#!pip install google-api-python-client
#!pip install google-cloud-dns

from google.oauth2 import service_account
from googleapiclient import discovery
from google.cloud import dns


#Downloaded credentials in JSON format
gcp_sa_credentials={
  "type": "service_account",
  "project_id": "mygcpprojectid-1122",
  "private_key_id": "myprivkeyid",
  "private_key": "-----BEGIN PRIVATE KEY-----\nmyprivatekey\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "myclientid",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/client-mail%40mygcpprojectid-1122.iam.gserviceaccount.com"
}

project_id=gcp_sa_credentials["project_id"]

credentials = service_account.Credentials.from_service_account_info(gcp_sa_credentials)
client = dns.Client(project=project_id,credentials=credentials)

edited Aug 27, 2023 at 7:38

answered Mar 20, 2022 at 8:31

Manju N

1,23213 silver badges18 bronze badges

2 Comments

mountrix Over a year ago

dns is not defined. We need pip install google-cloud-storage then from google.cloud import storage as dns

Manju N Over a year ago

@mountrix , I have specified callling dns.Client() now, can be managed through pip install google-cloud-dns then from google.cloud import dns

Brad · Accepted Answer · 2020-10-26 00:31:23Z

5

This is an old question however I want to add that you must create a new service account and not use an old one. A recent Google Cloud Next presentation on security stated that there is no guarantee that the default service account will exist in future, and it could be removed at any time (or its available permissions changed), so none of your applications should depend on it. Also I've found that there are potential issues with authentication when using the default service account and creating a new one is more likely to allow the control you need to successfully authenticate.

Refer to the following YouTube presentation from 11mins 10s in:

https://youtu.be/ZQHoC0cR6Qw?t=670

edited Oct 26, 2020 at 0:31

Brad

1802 silver badges10 bronze badges

answered Dec 20, 2018 at 21:19

Chris Halcrow

32.5k19 gold badges202 silver badges235 bronze badges

Collectives™ on Stack Overflow

How to Auth to Google Cloud using Service Account in Python?

5 Answers 5

4 Comments

3 Comments

2 Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

4 Comments

3 Comments

2 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related