Google Appengine JAVA -Zipping up blobstore files results in error 202 when saving back to blobstore

Question

I am working on a application in appengine that we want to be able to make the content available for offline users. This means we need to get all the used blobstore files and save them off for the offline user. I am using the server side to do this so that it is only done once, and not for every end user. I am using the task queue to run this process as it can easily time out. Assume all this code is running as a task.

Small collections work fine, but larger collections result in a appengine error 202 and it restarts the task again and again. Here is the sample code that comes from combination of Writing Zip Files to GAE Blobstore and following the advice for large zip files at Google Appengine JAVA - Zip lots of images saving in Blobstore by reopening the channel as needed. Also referenced AppEngine Error Code 202 - Task Queue as the error.

//Set up the zip file that will be saved to the blobstore
AppEngineFile assetFile = fileService.createNewBlobFile("application/zip", assetsZipName);
FileWriteChannel writeChannel = fileService.openWriteChannel(assetFile, true);
ZipOutputStream assetsZip = new ZipOutputStream(new BufferedOutputStream(Channels.newOutputStream(writeChannel)));

HashSet<String> blobsEntries = getAllBlobEntries(); //gets blobs that I need
saveBlobAssetsToZip(blobsEntries);

writeChannel.closeFinally(); 

.....

private void saveBlobAssetsToZip(blobsEntries) throws IOException  {        
    for (String blobId : blobsEntries) {

        /*gets the blobstote key that will result in the blobstore entry - ignore the bsmd as 
        that is internal to our wrapper for blobstore.*/

        BlobKey blobKey = new BlobKey(bsmd.getBlobId());

        //gets the blob file as a byte array    
        byte[] blobData = blobstoreService.fetchData(blobKey, 0, BlobstoreService.MAX_BLOB_FETCH_SIZE-1);
        String extension = type of file saved from our metadata (ie .jpg, .png, .pfd)

        assetsZip.putNextEntry(new ZipEntry(blobId + "." + extension));

        assetsZip.write(blobData);
        assetsZip.closeEntry();
        assetsZip.flush();

        /*I have found that if I don't close the channel and reopen it, I can get a IO exception
        because the files in the blobstore are too large, thus the write a file and then close and reopen*/

        assetsZip.close();
        writeChannel.close();
        String assetsPath = assetFile.getFullPath();
        assetFile = new AppEngineFile(assetsPath);
        writeChannel = fileService.openWriteChannel(assetFile, true);
        assetsZip = new ZipOutputStream(new BufferedOutputStream(Channels.newOutputStream(writeChannel)));

    }
}

What is the proper way to get this to run on appengine? Again small projects work fine and zip saves, but larger projects with more blob files results in this error.

Answer 1

I bet that the instance is running out of memory. Are you using appstats? It can consume a large amount of memory. If that doesn't work you will probably need to increase the instance size.