Search code examples
google-app-engineuploadcallbackgoogle-cloud-storageblobstore

Provide a callback URL in Google Cloud Storage signed URL

When uploading to GCS (Google Cloud Storage) using the BlobStore's createUploadURL function, I can provide a callback together with header data that will be POSTed to the callback URL.

There doesn't seem to be a way to do that with GCS's signed URL's

I know there is Object Change Notification but that won't allow the user to provide upload specific information in the header of a POST, the way it is possible with createUploadURL's callback.

My feeling is, if createUploadURL can do it, there must be a way to do it with signed URL's, but I can't find any documentation on it. I was wondering if anyone may know how createUploadURL achieves that callback calling behavior.

PS: I'm trying to move away from createUploadURL because of the __BlobInfo__ entities it creates, which for my specific use case I do not need, and somehow seem to be indelible and are wasting storage space.

Update: It worked! Here is how:

Short Answer: It cannot be done with PUT, but can be done with POST

Long Answer:

If you look at the signed-URL page, in front of HTTP_Verb, under Description, there is a subtle note that this page is only relevant to GET, HEAD, PUT, and DELETE, but POST is a completely different game. I had missed this, but it turned out to be very important.

There is a whole page of HTTP Headers that does not list an important header that can be used with POST; that header is success_action_redirect, as voscausa correctly answered.

In the POST page Google "strongly recommends" using PUT, unless dealing with form data. However, POST has a few nice features that PUT does not have. They may worry that POST gives us too many strings to hang ourselves with.

But I'd say it is totally worth dropping createUploadURL, and writing your own code to redirect to a callback. Here is how:

Code:

If you are working in Python voscausa's code is very helpful.

I'm using apejs to write javascript in a Java app, so my code looks like this:

            var exp = new Date()
            exp.setTime(exp.getTime() + 1000 * 60 * 100); //100 minutes

            json['GoogleAccessId'] = String(appIdentity.getServiceAccountName())
            json['key'] = keyGenerator()
            json['bucket'] = bucket
            json['Expires'] = exp.toISOString(); 
            json['success_action_redirect'] = "https://" + request.getServerName() + "/test2/";
            json['uri'] = 'https://' + bucket + '.storage.googleapis.com/'; 

            var policy = {'expiration': json.Expires
                        , 'conditions': [
                             ["starts-with", "$key", json.key],
                             {'Expires': json.Expires},
                             {'bucket': json.bucket},
                             {"success_action_redirect": json.success_action_redirect}
                           ]
                        };

            var plain = StringToBytes(JSON.stringify(policy))
            json['policy'] = String(Base64.encodeBase64String(plain))
            var result = appIdentity.signForApp(Base64.encodeBase64(plain, false));
            json['signature'] = String(Base64.encodeBase64String(result.getSignature()))

The code above first provides the relevant fields. Then creates a policy object. Then it stringify's the object and converts it into a byte array (you can use .getBytes in Java. I had to write a function for javascript). A base64 encoded version of this array, populates the policy field. Then it is signed using the appidentity package. Finally the signature is base64 encoded, and we are done.

On the client side, all members of the json object will be added to the Form, except the uri which is the form's address. 

        var formData = new FormData(document.forms.namedItem('upload'));
        var blob = new Blob([thedata], {type: 'application/json'})
        var keys = ['GoogleAccessId', 'key', 'bucket', 'Expires', 'success_action_redirect', 'policy', 'signature']
        for(field in keys)
          formData.append(keys[field], url[keys[field]])
        formData.append('file', blob)
        var rest = new XMLHttpRequest();
        rest.open('POST', url.uri)
        rest.onload = callback_function
        rest.send(formData)

If you do not provide a redirect, the response status will be 204 for success. But if you do redirect, the status will be 200. If you got 403 or 400 something about the signature or policy maybe wrong. Look at the responseText. If is often helpful.

A few things to note:

  • Both POST and PUT have a signature field, but these mean slightly different things. In case of POST, this is a signature of the policy.
  • PUT has a baseurl which contains the key (object name), but the URL used for POST may only include bucket name
  • PUT requires expiration as seconds from UNIX epoch, but POST wants it as an ISO string.
  • A PUT signature should be URL encoded (Java: by wrapping it with a URLEncoder.encode call). But for POST, Base64 encoding suffices.
  • By extension, for POST do Base64.encodeBase64String(result.getSignature()), and do not use the Base64.encodeBase64URLSafeString function
  • You cannot pass extra headers with the POST; only those listed in the POST page are allowed.
  • If you provide a URL for success_action_redirect, it will receive a GET with the key, bucket and eTag.
  • The other benefit of using POST is you can provide size limits. With PUT however, if a file breached your size restriction, you can only delete it after it was fully uploaded, even if it is multiple-tera-bytes.

What is wrong with createUploadURL?

The method above is a manual createUploadURL. But:

  • You don't get those __BlobInfo__ objects which create many indexes and are indelible. This irritates me as it wastes a lot of space (which reminds me of a separate issue: issue 4231. Please go give it a star)
  • You can provide your own object name, which helps create folders in your bucket.
  • You can provide different expiration dates for each link.

For the very very few javascript app-engineers:

function StringToBytes(sz) {
  map = function(x) {return x.charCodeAt(0)}
  return sz.split('').map(map)
}
Solution

  • You can include succes_action_redirect in a policy document when you use GCS post object.

    Docs here: Docs: https://cloud.google.com/storage/docs/xml-api/post-object
    Python example here: https://github.com/voscausa/appengine-gcs-upload

    Example callback result:

    def ok(self):
    """ GCS upload success callback """

    logging.debug('GCS upload result : %s' % self.request.query_string)
    bucket = self.request.get('bucket', default_value='')
    key = self.request.get('key', default_value='')
    key_parts = key.rsplit('/', 1)
    folder = key_parts[0] if len(key_parts) > 1 else None