Search code examples
pythongoogle-cloud-storageairflow

Setting ["GOOGLE_APPLICATION_CREDENTIALS"] from a dict rather than file path

I'm trying to set the environment variable from a dict but getting and error when connecting.

#service account pulls in airflow variable that contains the json dict with service_account credentials

service_account = Variable.get('google_cloud_credentials')

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]=str(service_account)

error

    PermissionDeniedError: Error executing an HTTP request: HTTP response code 403 with body '<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.</Details></Error>'

when reading if I use and point to file then there are no issues.

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]=/file/path/service_account.json

I'm wondering is there a way to convert the dict object to an os path like object? I don't want to store the json file on the container and airflow/google documentation isn't clear at all.

Solution

  • The Python stringio package lets you create a file-like object backed by a string, but that won't help here because the consumer of this environment variable is expecting a file path, not a file-like object. I don't think it's possible to do what you're trying to do. Is there a reason you don't want to just put the credentials in a file?