StorageException: Anonymous caller does not have storage.objects.get access

Question

When trying to run the below code on CircleCI

fun getJsonFromCloudStorage(): ByteArrayInputStream {
    val blobId = BlobId.of("my-company", "creds/my-company-creds.json")
    val storage = StorageOptions.getDefaultInstance().service
    val get = storage.get(blobId)
    return get.getContent().inputStream()
}

it will throw the below error during the integration tests.

> Task :test FAILED
function.GetMetadataFromYouTubeTest > extractIncorrectId FAILED
    java.lang.ExceptionInInitializerError
        at function.GetMetadataFromYouTube.expand(GetMetadataFromYouTube.kt:17)
        at function.GetMetadataFromYouTube.expand(GetMetadataFromYouTube.kt:14)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491)
        at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299)
        at function.GetMetadataFromYouTubeTest.extractIncorrectId(GetMetadataFromYouTubeTest.kt:71)

        Caused by:
        com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.get access to cni-analytics/creds/cni-awesome.json.
            at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:220)
            at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:414)
            at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:198)
            at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:195)
            at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
            at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
            at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
            at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:195)
            at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:209)
            at storage.CredentialHelper$Companion.getJsonFromCloudStorage(CredentialHelper.kt:18)
            at service.YoutubeService.initialiseYouTube(YoutubeService.kt:50)
            at service.YoutubeService.<init>(YoutubeService.kt:19)
            at MainKt.<clinit>(main.kt:15)
            ... 6 more

            Caused by:
            com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized
            {
              "code" : 401,
              "errors" : [ {
                "domain" : "global",
                "location" : "Authorization",
                "locationType" : "header",
                "message" : "Anonymous caller does not have storage.objects.get access to my-company/creds/my-company-creds.json.",
                "reason" : "required"
              } ],
              "message" : "Anonymous caller does not have storage.objects.get access to my-company/creds/my-company-creds.json."
            }
                at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
                at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
                at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:411)
                ... 17 more

I followed their documentation.

Answer 1

They said this in their documentation:

Note: To use certain services (like Google Cloud Datastore), you will also need to set the CircleCI $GOOGLE_APPLICATION_CREDENTIALS environment variable to ${HOME}/gcloud-service-key.json.

Instead I set $GOOGLE_APPLICATION_CREDENTIALS in the CircleCI UI to /home/circleci/gcloud-service-key.json and it worked.

I'm assuming this is because I was trying to reference an environment variable from the UI so ${HOME} had not been set when it was setting this environment variable. Perhaps if this environment variable was set in the config.yml ${HOME} would resolve.