Google Cloud Java SDK with Workload Identity?

Question

Trying to figure out how to authenticate with the storage API from within a GKE cluster.

Code:

Storage storage = StorageOptions.newBuilder()
  .setCredentials(ServiceAccountCredentials.getApplicationDefault())
  .setProjectId(gcpProjectId)
  .build().getService();

getApplicationDefault() is documented to use these means to authenticate with the API:

1. Credentials file pointed to by the {@code GOOGLE_APPLICATION_CREDENTIALS} environment variable
2. Credentials provided by the Google Cloud SDK {@code gcloud auth application-default login} command
3. Google App Engine built-in credentials
4. Google Cloud Shell built-in credentials
5. Google Compute Engine built-in credentials

The application is using the GCP workload identity feature, so the application (in-cluster) service account is annotated with:

serviceAccount.annotations.iam.gke.io/gcp-service-account: my-service-account@my-project.iam.gserviceaccount.com

Now the call to the storage account fails with the following error:

{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Primary: /namespaces/my-project.svc.id.goog with additional claims does not have storage.objects.create access to the Google Cloud Storage object.",
    "reason" : "forbidden"
  } ],
  "message" : "Primary: /namespaces/my-project.svc.id.goog with additional claims does not have storage.objects.create access to the Google Cloud Storage object."
}

This makes me think that the workload identity is not working correctly. I am expecting to receive an error message for my annotated service account and not the default one.

Is there anything else I should have been doing?

Answer 1

The answer, in part, aside from the annotation syntax, is that, just like me, you probably didn't look closely enough at this part in the documentation:

    gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]" \
  GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Notice the PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME] piece. It's something they give no examples on as far as syntax but it looks like this in my terraform.

resource "google_project_iam_member" "app-binding-2" {
  role   = "roles/iam.workloadIdentityUser"
  member = "serviceAccount:${local.ws_vars["project-id"]}.svc.id.goog[mynamespace/myk8ssaname]"
}

Weirdly you can bind this in the terraform even if the namespace doesn't exist, much less the service account. So you can run this first before deployments.