How can I upload a file larger than 32MB to GCS using the google-cloud client?

Question

I recently started using google-cloud client, and I made this little example (I have omitted/changed things for clarity's sake):

<form id="testform" method="post" enctype="multipart/form-data">
  <p>
    <div>
      <label for="fotos">Some file</label>
    </div>
    <div>
      <input type="file" name="testfile">
    </div>
  </p>
  <input type="submit" value="Submit">
</form>

from google.cloud import storage

class MyHandler(BaseHandler):
    def get(self):
        self.render("test.html")

    def post(self):
        file = self.request.POST["testfile"]
        filename = file.filename # + TimeStamp

        bucket = storage.Client().get_bucket(self.get_bucket_name())
        blob = bucket.blob(filename)

        blob.upload_from_string(data=file.file.read(),
                                content_type=file.type)

        self.write(blob.public_url)

This code works just fine with anything under 32MB, but it cannot handle larger files because they go through the app instead of directly to GCS. I have seen solutions using the Blobstore API, but those are old and the Blobstore isn't even used to store the data anymore (though the Blobstore API can still be used).

So I was wondering, is there a way to fix this using the google-cloud client? I have not seen a method to create an upload url in the docs.

Answer 1

Client should upload directly to GCS. A single connection from a GAE can't transfer more than 32 MB.

This shouldn't be a problem to you because you are already using a HTML form. https://cloud.google.com/storage/docs/xml-api/post-object

You may have to make the bucket publicly accessible depending on the access you'll need for your app. Else you'll have to specify GoogleAccessId in the PUT request generated on your backend and sent to frontend beforehand.

There is a small form upload example at the end of the page linked above.

Finally, I personally recommend to use AJAX to upload your file using the REST API https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload This will help you handle errors, redirects and success easier.