Localhost: how to get credentials to connect GAE Python 3 app and Datastore Emulator?

Question

I'd like to use the new Datastore Emulator together with a GAE Flask app on localhost. I want to run it in the Docker environment, but the error I get (DefaultCredentialsError) happens with or without Docker.

My Flask file looks like this (see the whole repository here on GitHub):

main.py:

from flask import Flask
from google.cloud import datastore


app = Flask(__name__)


@app.route("/")
def index():
    return "App Engine with Python 3"


@app.route("/message")
def message():
    # auth
    db = datastore.Client()

    # add object to db
    entity = datastore.Entity(key=db.key("Message"))
    message = {"message": "hello world"}
    entity.update(message)
    db.put(entity)

    # query from db
    obj = db.get(key=db.key("Message", entity.id))

    return "Message for you: {}".format(obj["message"])

The index() handler works fine, but the message() handler throws this error:

[2019-02-03 20:00:46,246] ERROR in app: Exception on /message [GET]
Traceback (most recent call last):
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/main.py", line 16, in message
    db = datastore.Client()
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/datastore/client.py", line 210, in __init__
    project=project, credentials=credentials, _http=_http
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/client.py", line 223, in __init__
    _ClientProjectMixin.__init__(self, project=project)
INFO     2019-02-03 20:00:46,260 module.py:861] default: "GET /message HTTP/1.1" 500 291
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/client.py", line 175, in __init__
    project = self._determine_default(project)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/datastore/client.py", line 228, in _determine_default
    return _determine_default_project(project)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/datastore/client.py", line 75, in _determine_default_project
    project = _base_default_project(project=project)
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/cloud/_helpers.py", line 186, in _determine_default_project
    _, project = google.auth.default()
  File "/tmp/tmpJcIw2U/lib/python3.5/site-packages/google/auth/_default.py", line 306, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

I checked the website in the error log and tried the JSON auth file (GOOGLE_APPLICATION_CREDENTIALS), but the result was that my app then connected with a production Datastore on Google Cloud, instead of the local Datastore Emulator.

Any idea how to resolve this?

Answer 1

I managed to solve this problem by adding env vars directly into the Python code (in this case in main.py) and using the Mock library:

import os

import mock
from flask import Flask, render_template, request
from google.cloud import datastore
import google.auth.credentials


app = Flask(__name__)

if os.getenv('GAE_ENV', '').startswith('standard'):
    # production
    db = datastore.Client()
else:
    # localhost
    os.environ["DATASTORE_DATASET"] = "test"
    os.environ["DATASTORE_EMULATOR_HOST"] = "localhost:8001"
    os.environ["DATASTORE_EMULATOR_HOST_PATH"] = "localhost:8001/datastore"
    os.environ["DATASTORE_HOST"] = "http://localhost:8001"
    os.environ["DATASTORE_PROJECT_ID"] = "test"

    credentials = mock.Mock(spec=google.auth.credentials.Credentials)
    db = datastore.Client(project="test", credentials=credentials)

The Datastore Emulator is then run like this:

gcloud beta emulators datastore start --no-legacy --data-dir=. --project test --host-port "localhost:8001"

Requirements needed:

Flask
google-cloud-datastore
mock
google-auth

GitHub example here: https://github.com/smartninja/gae-2nd-gen-examples/tree/master/simple-app-datastore