Create a connection between Python / Bigquery in a VM Instance Google Cloud (Problem with Credentials File)

Question

I'm creating a script that should run in a VM Instance (Google Cloud Plataform) and for that I use SSH (Linux) from this VM. This code will connect python with Bigquery.

I'm having problems with "credentials", when I'm running in my local machine, my credentials are in this file at my local directory and the code access this file and run the service. But when I run in SSH (VM Instance), the VM doesn't find the file, because it's not in VM directory.

What should I do to solve this problem? I think I need to put this file in VM Instance and take the new path, but I don't know how.

---

    from google.cloud import bigquery
    from google.oauth2 import service_account
    credentials = service_account.Credentials.from_service_account_file(
    r'C:\Users\Path\File.json')

    project_id = 'Project_ID523'
    client = bigquery.Client(credentials= credentials, project=project_id)

Answer 1

The best way to overcome your issue is to not keep the path to the service account file inside your source code.

• For accessing Bigquery locally, you need to set GOOGLE_APPLICATION_CREDENTIALS environment variable with value equal to the path to your service account file.
• The great part about it is that you don't have to do anything in your GCP VM because the bigquery client library can fetch the credentials automatically from the Compute Engine Instance's metadata. You just need to provide appropriate IAM permissions to the Compute Engine Service Account.

After setting the GOOGLE_APPLICATION_CREDENTIALS environment variable, your code will be transformed as below:

from google.cloud import bigquery
from google.oauth2 import service_account

project_id = 'Project_ID523'
client = bigquery.Client(project=project_id)

Note: No need to provide path to service account file in code as the client library will pick up the service account credentials from the environment variable.

Take a look at this link for more info.