Search code examples
authenticationgoogle-apigoogle-cloud-storagefile-transferdata-migration

Use Google Storage Transfer API to transfer data from external GCS into my GCS

I am working on a web application which comprises of ReactJs frontend and Java SpringBoot backend. This application would require users to upload data from their own Google Cloud storage into my Google Cloud Storage.

The application flow will be as follows -

  1. The frontend requests the user for read access on their storage. For this I have used oauth 2.0 access tokens as described here
  2. The generated Oauth token will be passed to the backend.
  3. The backend will also have credentials for my service account to allow it to access my Google Cloud APIs. I have created the service account with required permissions and generated the key using the instructions from here
  4. The backend will use the generated access token and my service account credentials to transfer the data.

In the final step, I want to create a transfer job using the google Storage-Transfer API. I am using the Java API client provided here for this.

I am having difficulty providing the authentication credentials to the transfer api. In my understanding, there are two different authentications required - one for reading the user's bucket and another for starting the transfer job and writing the data in my cloud storage. I haven't found any relevant documentation or working examples for my use-case. In all the given samples, it is always assumed that the same service account credentials will have access to both the source and sink buckets.

tl;dr Does the Google Storage Transfer API allow setting different source and target credentials for GCS to GCS transfers? If yes, how does one provide these credentials to the transfer job specification.

Any help is appreciated. Thanks!

Solution

  • This is not allowed for the the GCS Transfer API unfortunately, for this to work it would be required that the Service Account have access to both the source and the sink buckets, as you mentioned.

    You can try opening a feature request in Google's Issue Tracker if you'd like so that Google's Product Team can consider such a functionality for newer versions of the API, also you could mention that this is subject is not touched in the documentation, so it can be improved.