How to build a valid ActiveStorage direct upload request to GCS?

Question

This is about using ActiveStorage with GCS for an API client use case. (Rails 5.2.1, 5.2.2)

I'm writing a test to explore how to craft a request that mimics a direct upload to GCS, prepared by the generic DirectUploadsController. This generic controller is part of ActiveStorage. The idea is to later replicate the code in a mobile app talking to that same backend.

The AS configuration works well in development environment, using both upload via controllers, as well as direct upload using the JS integration that ships with AS. That's why I assume the config must be ok. ('test' and 'development' env use exact same setup at this stage.)

The test code is in the box below.

It always ends up raising a 403 Forbidden response from the RestClient.put call.

The response message complains about a signature mismatch, more details below. First the test code:

require 'test_helper'

class UploadControllerTest < ActionDispatch::IntegrationTest
  test "direct upload from controller prepared blob" do
    pathname = file_fixture('cube.png')
    data = pathname.binread
    content_type = "image/png"

    post rails_direct_uploads_path, params: {
      blob: {
        filename: pathname.basename,
        byte_size: pathname.size,
        checksum: Digest::MD5.base64digest(data),
        content_type: content_type
      }
    }

    assert_equal 27195, pathname.size
    assert_response :success

    json = response.parsed_body
    direct_upload = json["direct_upload"]
    signed_url    = direct_upload["url"]
    headers       = direct_upload["headers"]

    assert_equal({ "Content-MD5" => "tmBHZQCm+qBzGFEaDwmpnA==" }, headers)

    assert_match /&Signature=/, signed_url
    assert_match /&Expires=/, signed_url
    assert_match %r{^https://storage.googleapis.com}, signed_url

    response = RestClient.put(
      signed_url,
      data,
      headers.merge("Content-Type" => content_type)
    )

    assert_response :success
  rescue RestClient::Forbidden => e
    pp e.response.body
    fail "Failing with 403 Forbidden" # always ends up here
  end
end

The resulting response body is this XML:

  <?xml version='1.0' encoding='UTF-8'?>
  <Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>
      The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
    </Message>
    <StringToSign>PUT\n" +
    "tmBHZQCm+qBzGFEaDwmpnA==\n" +
    "image/png\n" +
    "1544517548\n" +
  "/planprop-test-bucket/gVn9zVCumGJxiu2kU6mFWUVV</StringToSign>
  </Error>

The error code is:

SignatureDoesNotMatch

and the accompanying message this:

The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

The listed parts of the signature string are checksum (asserted above), expiration time (part of the URL), content type (asserted above), and the object (bucket name and key, part of the URL). So I don't see a part where a mismatch might sneak in.

What is amiss?

Answer 1

The problem with the code above is that a Content-Type header of application/x-www-form-urlencoded is sent, while none is used when signing.

To make it work change the PUT request code to

response = RestClient.put(
  signed_url,
  data,
  headers.merge(content_type: "")
)

Using nil will still make this default show up in the request.

This behavior isn't unique for RestClient, btw. but the same across a number of Ruby http clients I tested (Faraday, net/http, httpclient). Excon is an exception here, not sending a default content type header without being told to do so.