GKE / Cloud IAM workload Identity setup error 403

Question

We have setup a CloudSQL proxy as a sidecar container for one of our Java / Tomcat based app.

Here's how we setup workload identity to enable our app to connect to CloudSQL through cloudsql proxy:

1. Created Cloud IAM Service Account and gave it SQL Client permission:

2. Setup Policy Binding as follows:

   gcloud iam service-accounts add-iam-policy-binding \
     --role roles/iam.workloadIdentityUser \
     --member "serviceAccount:[PROJECT_ID].svc.id.goog[default/default]" \
    [GSA_NAME]@[PROJECT_ID].iam.gserviceaccount.com
   
   

3. Added annotation to GKE Service Account:

   kubectl annotate serviceaccount \
     --namespace [K8S_NAMESPACE] \
     [KSA_NAME] \
     iam.gke.io/gcp-service-account=[GSA_NAME]@[PROJECT_ID].iam.gserviceaccount.com
   

4. But when we test this using:

   kubectl run --rm -it \
     --generator=run-pod/v1 \
     --image google/cloud-sdk:slim \
     --serviceaccount [KSA_NAME] \
     --namespace [K8S_NAMESPACE] \
     workload-identity-test
   

Despite doing everything correctly as explained on this page still results in:

Error 403: The client is not authorized to make this request., notAuthorized

Answer 1

It turns out there is a glitch in Google Cloud IAM which seems to affect service accounts.

1. Delete the existing Cloud IAM service account created in step 1 and role associated with it.
2. Recreate the service account again (with exam same permissions)

which should fix the issue. Of course you'll have to redo step 2 and 3 again to complete the setup but it works.