Search code examples
javagoogle-cloud-platformgoogle-cloud-storage

Getting 403 from GCP metadata server

I'm using GCP storage to upload a file and return it's signed/public URL.

I'm getting below error when trying to signUrl after uploading my file to gcp storage.

Caused by: java.io.IOException: Unexpected Error code 403 trying to get service accounts from Compute Engine metadata: This metadata endpoint is concealed for ?recursive calls

    at com.google.auth.oauth2.ComputeEngineCredentials.getDefaultServiceAccount(ComputeEngineCredentials.java:383)
    at com.google.auth.oauth2.ComputeEngineCredentials.getAccount(ComputeEngineCredentials.java:336)
    ... 18 common frames omitted

This is specifically when service-account call goes to gcp metadata.

com.google.auth.oauth2.ComputeEngineCredentials#getDefaultServiceAccount

HttpResponse response = getMetadataResponse(getServiceAccountsUrl());

My GCP java library version is 1.80.0.

Not able to find this error anywhere.

Let me know if any further details are required here.

EDIT 1:

This is a minimal version of the actual code, gist of what is happening.

int ttlInMinutes = 5;
String objectName = "object-name";
String bucket = "some-bucket";
Storage storage = StorageOptions.getDefaultInstance().getService();
BlobInfo blobinfo = BlobInfo.newBuilder(BlobId.of(bucket, objectName)).build();
URL url = storage.signUrl(blobinfo, ttlInMinutes, TimeUnit.MINUTES, withV4Signature());

At this point, I get the mentioned error.

Solution

  • We are using GCE with k8s, where we are deploying pods on GCE servers, and service accounts are linked with server that the pod is deployed on, and not directly with the pod itself.

    The issue was that in k8s yaml file, used to deploy the pod, we were not setting hostNetwork. After setting hostNetwork to true the metadata request returned result successfully.

    Before this setting, I guess the pod will run in it's own network instead of using the server's network. More info: What is hostNetwork in Kubernetes?