Search code examples
google-app-engineblobstore

constant HTTP 503 error when uploading files to Google App Engine Blobstore (and High Replication Datastore)


I have an application which uploads files to Google App Engine. Upload works OK from most computers, but on one system (at different geographical location) we constantly get HTTP 503 errors when uploading files.

The app follows the App Engine documentation. We first create upload URL with BlobstoreService.createUploadUrl, return the URL to client, and then client submits files to that URL.

What is the reason of that 503 error and how to solve it? Strange thing that it constantly reproduces on that system, while the same app successfully uploads files from other systems.

In the log below you can see: 1. HTTP headers of request/response for upload URL, 2. The upload URL returned 3. HTTP headers of request/response to that upload URL.

GET /upload-url HTTP/1.1
Host: cl-test-grid.appspot.com
User-Agent: Drakma/1.2.6 (SBCL 1.0.57; Darwin; 10.8.0; http://weitz.de/drakma/)
Accept: */*
Connection: close

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Vary: Accept-Encoding
Date: Sun, 27 May 2012 05:30:03 GMT
Server: Google Frontend
Cache-Control: private
Connection: close

2012-05-27 05:30:02 upload-url: http://cl-test-grid.appspot.com/_ah/upload/AMmfu6b0LpFQeYiI5sZa6KcBy9_JpIreCEjYY-FEbd3b8wPApdW3_rUIETK-geaaw_x5xq3QHhcn_wd2Pykos07xIRJTZesBIX0jkI3L4-Ri3TiOqnbS3iM/ALBNUaYAAAAAT8G-M8FF5sO4mMkXVzZX8hTcUv0OQlIK/

POST /_ah/upload/AMmfu6b0LpFQeYiI5sZa6KcBy9_JpIreCEjYY-FEbd3b8wPApdW3_rUIETK-geaaw_x5xq3QHhcn_wd2Pykos07xIRJTZesBIX0jkI3L4-Ri3TiOqnbS3iM/ALBNUaYAAAAAT8G-M8FF5sO4mMkXVzZX8hTcUv0OQlIK/ HTTP/1.1
Host: cl-test-grid.appspot.com
User-Agent: Drakma/1.2.6 (SBCL 1.0.57; Darwin; 10.8.0; http://weitz.de/drakma/)
Accept: */*
Connection: close
Content-Type: multipart/form-data; boundary=----------6mlF09BtSz1OnI06GN22cqE9jBOi8L2ozKYqrT7fQy8dDDhwk2
Content-Length: 342917

HTTP/1.1 503 Service Unavailable
Server: HTTP Upload Server Built on May 23 2012 15:51:04 (1337813464)
Date: Sun, 27 May 2012 05:30:15 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate
Content-Length: 0
Content-Type: text/html
Connection: close

Solution

  • We too have been seeing a LOT of 503 errors during blobstore uploads. This has been from customer systems as well as in houses systems. As it was working fine until recently, and we haven't changed any related code, I can only assume that there is a problem at Google's end.

    I couldn't find Stefano's report on the issue tracker, so I logged a new issue, number 7619:

    http://code.google.com/p/googleappengine/issues/detail?id=7619

    Please star this if you're still having this issue to encourage Google to look into it and either fix it or advise on how to detect and work around the error.