Search code examples
javagoogle-app-enginezipgoogle-cloud-storagelarge-files

How to download a large Zip from GAE

I used the methods suggested here: Creating ZIP archives in Google App Engine ( Java)
And here How to create a zip archive containing google cloud storage objects within appengine java app? to return a zip file, the problem I'm having is that the response is bigger than the allowed ~30M. What is the best practice to deal with it?

Solution

  • Most optimal way will be saving to Storage Bucket, and then serve it from there.

    It also allows you to:

    • prepare file from a TaskQueue job, which is not limited to 60 seconds so you can process much more data, handle errors, etc
    • give user ability to pause/resume download, use a download manager, etc, w/o forcing your frontend to start the job from the beginning each time

    Basically i'm suggesting:

    1. give user an TicketId for backend job
    2. send task to TaskQueue
    3. prepare data in this backend job, upload to Storage Bucket, set TicketId as done
    4. browser waits until TicketId is marked as done (AJAX of Channel API)
    5. then a special servlet gives this file, from a Storage

    There're two options to serve this file later:

    (optimal) Give a link directly to Storage object:

    or by using Blobstore:

    BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
BlobKey blobKey = blobstoreService.createGsBlobKey("/gs/<bucket>/<object>");
blobstoreService.serve(blobKey, resp); //where resp is your HttpServletResponse