Not able to successfully upload files using signed URL to google cloud Storage from Advanced REST Client

Question

I am trying to create a signed URL and upload files from my PC to google cloud storage using it.

I am using Advanced REST Client(ARC) as the client side application. On the server side, I have a jersey based server running on Appengine.

I first send a GET request from ARC, on receiving which the app engine generates a signed URL and returns it back in the response.

After that I do a PUT request with the file I want to upload in the body and the request URL set to what was received in the response to GET.

The code snippet to create signed URL:

        String encodedUrl = null;
        String contentMD5 = "";
        String contentType = "";
        String httpVerb;


        httpVerb = "PUT";

        Calendar calendar = Calendar.getInstance();
        calendar.add(Calendar.MINUTE, 10);
        long expiration = calendar.getTimeInMillis() / 1000L;

        String canonicalizedResource = "/" + bucketName + "/" + objectName;
        String baseURL = "https://storage.googleapis.com" + canonicalizedResource;

        String stringToSign =
                httpVerb + "\n" + contentMD5 + "\n" + contentType + "\n" + expiration + "\n"
                        + canonicalizedResource;

        AppIdentityService service = AppIdentityServiceFactory.getAppIdentityService();
        String googleAccessId = service.getServiceAccountName();

        SigningResult signingResult = service.signForApp(stringToSign.getBytes());
        String encodedSignature = null;
        try {
            encodedSignature =
                    new String(Base64.encodeBase64(signingResult.getSignature(), false), "UTF-8");
        } catch (UnsupportedEncodingException e) {
            throw new InternalServerErrorException();
        }

        String signature = null;
        try {
            signature = URLEncoder.encode(encodedSignature, "UTF-8").toString();
        } catch (UnsupportedEncodingException e) {
            throw new InternalServerErrorException();
        }

        encodedUrl =
                baseURL + "?GoogleAccessId=" + googleAccessId + "&Expires=" + expiration
                        + "&Signature=" + signature;
        System.out.println("Signed URL is: "+encodedUrl);

However I observe the following issue:

1. Whenever I send the PUT request with any file type, I get the following error:

   Error - 403
   Code - SignatureDoesNotMatch

   Message - The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method

Please note that in my code, I am setting the content Type to "" while creating the string to sign. Also while creating the PUT request I don't include any Content-type header.

As far as I understand, if I don't include the contentType in the stringToSign while creating the signed URL and also not add it as a header while sending PUT request it should be fine. So what could be the reason for the error?

2. After that I changed by code and added the contentType while creating the stringToSign in the code and also gave the corresponding Content-Type header while sending the PUT request.

In this case I am able to upload the file, however the uploaded file is modified/corrupted.I tried with text/plain and image/jpeg.

The problem is that the following text is added at the beginning of the file:

------WebKitFormBoundaryZX8rPPhnm1WXPrUf
Content-Disposition: form-data; name="fileUpload5"; filename="blob"
Content-Type: text/plain

I can see this in the text file and on opening the .jpg file in the hex editor. The .jpg does not open in standard image application since the file has been corrupted by the text in the beginning

Am I missing something here? Is this any issue in the Advanced REST Client? Actually whenever I send a PUT request with some file in the body, I get a message in the ARC saying that : The content-type header will be finally changed to multipart/form-data while sending the request However, I saved exported all the messages to a file from ARC and I didn't find any message with Content-type header set to multipart/form-data. So why does this message come and is it actually an issue?

Answer 1

URL-signing code is tricky and notoriously hard to debug. Fortunately, Google's google-cloud library has a signUrl function that takes care of this for you. I highly encourage you to use it instead of rewriting it yourself. Here's the documentation.

Now, if you want to debug it yourself, checking the error message is super useful. It will include a complete copy of the string the server checked the signature of. Print out your stringToSign variable and see how it's different. That'll tell you what's wrong.

Now, on to your specific problem: it sounds like you are generating an acceptable signed URL, but then your client is attempting to upload to GCS as if it were doing a multipart, form upload. The text you're looking at is part of an HTTP multipart request, and the "multipart/form-data" warning also points in that direction. See if the app you're using has some sort of "Form" mode/option that you are perhaps accidentally using?