I'm serving some files from GCS through a GAE application (using flask), recently I hit the 32 MB limit. It should according to the documentation be possible to serve files from GCS that are larger than 32 MB, but I can't find the correct python3 apis.
Response limits Dynamic responses are limited to 32MB. If a script handler generates a response larger than this limit, the server sends back an empty response with a 500 Internal Server Error status code. This limitation does not apply to responses that serve data from the Blobstore or Cloud Storage.
I've found the blobstore API which can serve files from GCS, but unfortunately that library is deprecated and not python3 compatible. This is what I think the code would have looked like in that approach
blobstore_filename = '/gs/{}/{}'.format(CLOUD_STORAGE_BUCKET, path)
blob_key = blobstore.create_gs_key(blobstore_filename)
response = Response("This should be overriden by App Enginge")
response.headers['Access-Control-Allow-Origin'] = '*'
response.headers['X-AppEngine-BlobKey'] = blob_key
return response
I think in PHP there is a specific API method that does exactly what I need CloudStorageTools#serve
I've seen some suggestions for similar problems to use Signed URLs but that seems more like a work-around rather than an actual solution to the problem.
I had expected to find a method to serve content from GCS, I don't know if my search skills are failing me or the method I need doesn't exist for the python3 environment.
You can have a route on your app that returns a redirect to the GCS Signed URL.
This way you can control the access to the route, but the content itself is not served by your app.
Serving through your app will increase your cost and lower your performance, since a dynamic request is not cached by default and AppEngine instance time and egressing data are more expensive than serving from GCS.
The App Engine standard environments that have this feature, like PHP and Python 2, does this redirect internally, so the client do not see the access to GCS, but it is happening similarly.
None of the AppEngine specific APIs are available on Python 3 standard, see here.
In order to create a Signed URL on App Engine standard Python 3 you'll need to create a Service Account key manually. The Application Default Credentials provided on Compute Engine and App Engine cannot be used for digital signing. Python 2, Java, Go and PHP allow it because they use the internal App Identity API to generate the signature.
Although those are not available yet, Google's only remark on this till now that I have seen is:
At this time, App Engine APIs are not available in the Python 3.7 runtime
So we can still have some hope.