Search code examples
python-3.xgoogle-app-enginepycharm

How to test/run/debug Non-Legacy App Engine project using PyCharm on Windows

For years I have been developing for Python 2.7 App Engine using PyCharm on Windows. Works amazingly well, I can run or debug locally at the press of a single button within PyCharm. Love it!

As part of the forced migration to Python 3 App Engine, this beautiful combination no longer apparently works. Google docs say that its dev_appserver, which PyCharm runs behind the scenes, does not support development of Python 3 apps on Windows. NoCommandLine has developed a patch that may make this possible, but it’s not clear to me how robust this is, or how sustainable (assuming it works). Even if this would work, would PyCharm correctly call dev_appserver? PyCharm's docs say they always use the Python 2.7 runtime when creating a new project. It's not clear what this means, but right away the project complains about missing app.yaml keywords (which were previously required, and are now no longer permitted).

I would love to continue the single-button run/debug local testing using PyCharm. Is that possible?

I'm starting to think the best approach to development and local testing is to create a Flask project within PyCharm, and use PyCharm to run/debug using the Flask development server, separately running a local App Engine datastore emulator so it can access a local NDB. One "minor" issue with this is that I'd have to port numerous app.yaml static handlers to Flask to actually test a localhost web page.

Is there a better way? Please help me from having to rediscover the wheel...

Solution

  • So you want to develop for Python 3 App Engine on Windows using PyCharm? Here's how it's done (order matters!)

    1. Install a copy of the Google Cloud SDK for Python
    2. Run gcloud components install app-engine-python and app-engine-python-extras
    3. set CLOUDSDK_PYTHON=C:\Python\Python39\python.exe (or wherever your Python lives)
    4. Using PyCharm create a Flask project, and set its run configuration to launch app/main.py (you will simply be running/debugging a Flask app, ignoring app.yaml completely for local dev)
    5. In your main.py, you may need to duplicate any static handlers you have in your app.yaml
    6. Copy the google folder from the App Engine SDK installation (e.g., C:\Users\XXX\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google) into your project's venv\Lib\site-packages (this will allow PyCharm to find ALL the google modules, since otherwise you will likely have 2 google module installations as described in the question)
    7. Allow PyCharm to satisfy the requirements.txt appengine-python-standard>=1.0.0, installing more libraries into your project's venv, and overwriting some of the files you copied
    8. In your project's venv\Lib\site-packages\google\appengine\runtime\thread_hooks.py, comment out the line importlib.reload(threading_module) so that PyCharm debugging will work

    To use a local datastore, use the datastore emulator:

    1. Include google-cloud-ndb in your requirements.txt
    2. gcloud components install beta
    3. You will also need a Java JRE (runtime engine) v11+ in your PATH
    4. In your project's Run/Debug configuration, set environment variable DATASTORE_EMULATOR_HOST=localhost:8081 (or whatever port you are using)
    5. Start the emulator via gcloud beta emulators datastore start before you run/debug your program from PyCharm

    You can now develop and debug as normal... but you cannot locally access legacy bundled services.

    If you want to do that, you will need to use dev_appserver, which is not officially supported on Windows. NoCommandLine has developed a patch that allows this, but while the patch will allow you to run, it will not allow you to debug within PyCharm. The dev_appserver also takes a LONG time to startup, even with the --python_virtualenv_path.