Transfer From S3 to Google Storage - Incorrect Key

Question

I've been trying for past couple of hours to setup a transfer from S3 to my google storage bucket.

The error that i keep getting, when creating the transfer is: "Invalid access key. Make sure the access key for your S3 bucket is correct, or set the bucket permissions to Grant Everyone."

Both the access key and the secret are correct, given that they are currently in use in production for S3 full access.

Couple of things to note:

1. CORS-enabled on S3 bucket
2. Bucket policy only allows authenticated AWS users to list/view its contents
3. S3 requires signed URLs for access

Bucket Policy:

{
    "Version": "2008-10-17",
    "Id": "Policy234234234",
    "Statement": [
        {
            "Sid": "Stmt234234",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetObjectAcl",
                "s3:RestoreObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:PutObjectVersionAcl",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:PutObject",
                "s3:GetObjectVersionAcl"
            ],
            "Resource": "arn:aws:s3:::mybucket/*"
        },
        {
            "Sid": "2",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity xyzmatey"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::mybucket/*"
        },
        {
            "Sid": "3",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::mybucket"
        }
    ]
}

CORS Policy

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>http://www.mywebsite.com</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>DELETE</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedHeader>AUTHORIZATION</AllowedHeader>
    </CORSRule>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>HEAD</AllowedMethod>
        <AllowedHeader>AUTHORIZATION</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Any idea where i have gone wrong?

EDIT: I've setup the gsutil tool on a google compute instance and did a copy with the same AWS keys on the exact bucket. Worked like a charm..

Answer 1

I am an engineer on Transfer service. The reason you encountered this problem is that AWS S3 region ap-southeast-1 (Singapore) is not yet supported by the Transfer service, because GCP does not have networking arrangement with AWS S3 in that region. We can consider to support that region now but your transfer will be much slower than other regions.

On our end, we are making a fix to display a clearer error message.