How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)

Question

What's the best strategy for managing third-party Python libraries with Google App Engine?

Say I want to use Flask, a webapp framework. A blog entry says to do this, which doesn't seem right:

$ cd /tmp/
$ wget http://pypi.python.org/packages/source/F/Flask/Flask-0.6.1.tar.gz
$ tar zxf Flask-0.6.1.tar.gz
$ cp -r Flask-0.6.1/flask ~/path/to/project/
(... repeat for other packages ...)

There must be a better way to manage third-party code, especially if I want to track versions, test upgrades or if two libraries share a subdirectory. I know that Python can import modules from zipfiles and that pip can work with a wonderful REQUIREMENTS file, and I've seen that pip has a zip command for use with GAE.

(Note: There's a handful of similar questions — 1, 2, 3, 4, 5 — but they're case-specific and don't really answer my question.)

Answer 1

(Jun 2021) This post is over a decade old, and so an updated answer is warranted now.

1. Python 3: list 3P libraries in requirements.txt along with any desired version#s; they'll be automatically installed by Google upon deployment. (This is the same technique used if you decide to migrate your app to Google Cloud Functions or Cloud Run.)
2. Python 2 without built-in 3P libraries (regular 3P libraries):

• Create requirements.txt as above
• Install/self-bundle/copy them locally, say to lib, via pip install -t lib -r requirements.txt
• Create appengine_config.py as shown in step 5 on this page

3. Python 2 with built-in 3P libraries (special set of 3P libraries):

• All listed 3P libraries linked above are "built-in," meaning they're available on App Engine servers so you don't have to copy/self-bundle them w/your app (like in #2 above)
• It suffices to list them with an available version in the libraries: section of your app.yaml like this
• (Don't put built-in libraries in requirements.txt nor use pip install to install them locally unless you want to self-bundle because, say if you need a newer version of the built-in library.)
• Create appengine_config.py like the above.

If you have a Python 2 app with both built-in and non-built-in 3P libraries, use the techniques in both #2 and #3 above (built-in libraries in app.yaml and non-built-in libraries in requirements.txt and run the pip install cmd above). One of the improvements in the second generation runtimes like Python 3 is that all these games with 3P libraries go away magically (see #1 above).

Example: Flask

Flask is a 3rd-party micro web framework, and it's an interesting case for this specific question. For Python 3, they all go into requirements.txt, so you'd just add flask to that file, and you're done. (Just deploy from there.)

For Python 2, it's even more interesting because it's a built-in library. Unfortunately, the version on App Engine servers is 0.12. Who wants to use that when we're at/beyond 2.0.3 now?!? So instead of putting it in app.yaml like other built-in libraries, you'd pretend the built-in version doesn't exist and put it in requirements.txt then run pip2 install -t lib -r requirements.txt to bundle/vendor it with your application code. (However, the final version for Python 2 is 1.1.4, so that's what gets installed.)