I'm having trouble uploading images to google cloud storage in newly created app engine apps using the default bucket.
I'm using upload code that works perfectly fine when deployed to older app engine apps that have already active default buckets. When I create a new app engine project or activate the default gcs bucket on appengine.google.com then I get a strange 500 (503) error.
my upload code uses the cloudstorage client and looks like this:
def save_to_gcs(file_obj, obj):
# pass a file and a datastore entity to associate the file to
serving_url = ''#just assign it adn reassign later
time_stamp = int(time.time())
app_id = app_identity.get_application_id()
logging.error("got app_id .... %s" % app_id)
fname = '/%s.appspot.com/post_%s_%s.jpg' % (app_id, obj.key.id(), time_stamp)
# logging.error(file_obj)
logging.error("fname .... %s" % fname)
gcs_file = gcs.open(fname, 'w', content_type="image/jpeg")
gcs_file.write(file_obj)
logging.error("written file obj ....")# <-- this is where the logs stop
gcs_file.close()
gcs_filename = "/gs%s" % fname
serving_url = images.get_serving_url(blobstore.create_gs_key(gcs_filename))
logging.error("got serving url ....")
media_obj = save_gcs_to_media(gcs_filename, serving_url, obj)
logging.error("saved media obj ....")
return media_obj
I think the errors have something to do with the file.close() method and probably some auth thing with the default bucket.
I'm properly authenticated and the file I'm uploading is I about 100kb, so nothing large.
Any help would be really appreciated (in fact it would prevent my hair first going grey, then falling out).
Sorry for the mess, but here are the logs:
E 11:05:54.412 - - - - COVER - - - -
E 11:05:54.413 True
E 11:05:54.413 got time stamp
E 11:05:54.413 got time file name
E 11:05:54.413 got app_id .... emile-test
E 11:05:54.413 fname .... /emile-test.appspot.com/post_5629499534213120_1436432754.jpg
E 11:05:54.696 written file obj ....
D 11:05:55.025 Tasklet is <bound method _StorageApi.urlfetch_async of <cloudstorage.storage_api._StorageApi object at 0xfbd7bf10>>
D 11:05:55.025 Got result <google.appengine.api.urlfetch._URLFetchResult object at 0xfbd9a810> from tasklet.
D 11:05:55.025 Retry in 0.1 seconds.
D 11:05:55.477 Got result <google.appengine.api.urlfetch._URLFetchResult object at 0xfbd9a9b0> from tasklet.
D 11:05:55.478 Retry in 0.2 seconds.
D 11:05:56.152 Got result <google.appengine.api.urlfetch._URLFetchResult object at 0xfbd9ab30> from tasklet.
D 11:05:56.153 Retry in 0.4 seconds.
D 11:05:57.244 Got result <google.appengine.api.urlfetch._URLFetchResult object at 0xfbd9acb0> from tasklet.
D 11:05:57.245 Retry in 0.8 seconds.
D 11:05:58.435 Got result <google.appengine.api.urlfetch._URLFetchResult object at 0xfbd7b690> from tasklet.
D 11:05:58.435 Retry in 1.6 seconds.
D 11:06:00.542 Got result <google.appengine.api.urlfetch._URLFetchResult object at 0xfe4ce630> from tasklet.
D 11:06:00.542 Retry in 3.2 seconds.
D 11:06:04.179 Tasklet failed after 7 attempts and 9.48189997673 seconds in total
**E 11:06:04.180 Expect status [200] from Google Storage. But got status 503.**
Path: '/emile-test.appspot.com/post_5629499534213120_1436432754.jpg'.
Request headers: {'content-range': 'bytes 0-108067/108068', 'x-goog-api-version': '2', 'accept-encoding': 'gzip, *'}.
Response headers: {'content-length': '0', 'x-guploader-uploadid': 'AEnB2UqzyZwtoNSy8sTEUq7a8fai-A5aDVlMccCj1UVEo72NAUUw4kyDotehcf8RDusUzm52092rdO0CR9JxIC9ba1J5Kj1xuw'}.
Body: ''.
Extra info: {'upload_path': '/emile-test.appspot.com/post_5629499534213120_1436432754.jpg?upload_id=AEnB2UqzyZwtoNSy8sTEUq7a8fai-A5aDVlMccCj1UVEo72NAUUw4kyDotehcf8RDusUzm52092rdO0CR9JxIC9ba1J5Kj1xuw'}.
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~emile-test/1.385589451941224330/main.py", line 757, in post
media_obj = utils.save_to_gcs(image, post)
File "/base/data/home/apps/s~emile-test/1.385589451941224330/utils.py", line 131, in save_to_gcs
gcs_file.close()
**File "/base/data/home/apps/s~emile-test/1.385589451941224330/cloudstorage/storage_api.py", line 750, in close**
self._flush(finish=True)
File "/base/data/home/apps/s~emile-test/1.385589451941224330/cloudstorage/storage_api.py", line 804, in _flush
self._send_data(data, self._written, file_len)
File "/base/data/home/apps/s~emile-test/1.385589451941224330/cloudstorage/storage_api.py", line 837, in _send_data
{'upload_path': self._path_with_token})
File "/base/data/home/apps/s~emile-test/1.385589451941224330/cloudstorage/errors.py", line 141, in check_status
raise ServerError(msg)
ServerError: Expect status [200] from Google Storage. But got status 503.
Path: '/emile-test.appspot.com/post_5629499534213120_1436432754.jpg'.
Request headers: {'content-range': 'bytes 0-108067/108068', 'x-goog-api-version': '2', 'accept-encoding': 'gzip, *'}.
Response headers: {'content-length': '0', 'x-guploader-uploadid': 'AEnB2UqzyZwtoNSy8sTEUq7a8fai-A5aDVlMccCj1UVEo72NAUUw4kyDotehcf8RDusUzm52092rdO0CR9JxIC9ba1J5Kj1xuw'}.
Body: ''.
Extra info: {'upload_path': '/emile-test.appspot.com/post_5629499534213120_1436432754.jpg?upload_id=AEnB2UqzyZwtoNSy8sTEUq7a8fai-A5aDVlMccCj1UVEo72NAUUw4kyDotehcf8RDusUzm52092rdO0CR9JxIC9ba1J5Kj1xuw'}.
I fixed exactly this problem (identical stack trace) in my own GAE project today (I'm migrating from the Files API in blobstore to cloudstorage). The steps I took were:
went to the Google Developers Console
https://console.developers.google.com/project/YOUR_APP_ID/storage/browser
where YOUR_APP_ID is the name the app without the .appspot.com suffix, e.g. 'foo', not 'foo.appspot.com',
Clicked on the three vertical dots on the right of my project
Selected Bucket Permissions
Added my email address as Entity=User, name=myemail@gmail.com, Access=Owner.
I'm not 100% sure why this fixed the problem, but it's done the trick.