I'm trying to set up Cloud SQL Proxy running as a sidecar in my GKE cluster. The configuration is done via Terraform. I've set up workload identity, required service accounts, and so on. When launching ./cloud_sql_proxy from within the GKE cluster (kubectl run -it --image google/cloud-sdk:slim --serviceaccount ksa-name --namespace k8s-namespace workload-identity-test), I get the following output:
root@workload-identity-test:/# ./cloud_sql_proxy -instances=project-id:europe-west4:db-instance=tcp:5432
2020/11/24 17:18:39 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/11/24 17:18:40 GcloudConfig: error reading config: exit status 1; stderr was:
ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Unable to generate access token; IAM returned 404 Not Found: Requested entity was not found.\\n'", <google_auth_httplib2._Response object at 0x7fc5575545f8>)
Please run:
$ gcloud auth login
to obtain new credentials.
If you have already logged in with a different account:
$ gcloud config set account ACCOUNT
to select an already authenticated account to use.
2020/11/24 17:18:41 GcloudConfig: error reading config: exit status 1; stderr was:
ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Unable to generate access token; IAM returned 404 Not Found: Requested entity was not found.\\n'", <google_auth_httplib2._Response object at 0x7f06f72f45c0>)
Please run:
$ gcloud auth login
to obtain new credentials.
If you have already logged in with a different account:
$ gcloud config set account ACCOUNT
to select an already authenticated account to use.
2020/11/24 17:18:41 errors parsing config:
Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/project-id/instances/europe-west4~db-instance?alt=json&prettyPrint=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.admin" not defined
Here's troubleshooting I've done so far:
root@workload-identity-test:/# gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* [email protected]
To set the active account, run:
$ gcloud config set account `ACCOUNT`
λ gcloud container clusters describe mycluster --format="value(workloadIdentityConfig.workloadPool)"
project-id.svc.id.goog
λ gcloud container node-pools describe mycluster-node-pool --cluster=mycluster --format="value(config.workloadMetadataConfig.mode)"
GKE_METADATA
λ gcloud container node-pools describe mycluster-node-pool --cluster=mycluster--format="value(config.oauthScopes)"
https://www.googleapis.com/auth/monitoring;https://www.googleapis.com/auth/devstorage.read_only;https://www.googleapis.com/auth/logging.write;https://www.googleapis.com/auth/cloud-platform;https://www.googleapis.com/auth/userinfo.email;https://www.googleapis.com/auth/compute;https://www.googleapis.com/auth/sqlservice.admin
λ kubectl describe serviceaccount --namespace k8s-namespace ksa-name
Name: ksa-name
Namespace: k8s-namespace
Labels: <none>
Annotations: iam.gke.io/gcp-service-account: [email protected]
Image pull secrets: <none>
Mountable secrets: ksa-name-token-87n4t
Tokens: ksa-name-token-87n4t
Events: <none>
λ gcloud iam service-accounts get-iam-policy [email protected]
bindings:
- members:
- serviceAccount:project-id.svc.id.goog[k8s-namespace/ksa-name]
role: roles/iam.workloadIdentityUser
etag: BwW02zludbY=
version: 1
λ kubectl get networkpolicy --namespace k8s-namespace
No resources found in k8s-namespace namespace.
λ gcloud projects get-iam-policy project-id
bindings:
- members:
- serviceAccount:[email protected]
role: roles/cloudsql.editor
Expected result (I got this running on another cluster and changed configuration afterwards, can't find where my mistake is):
root@workload-identity-test:~# ./cloud_sql_proxy -instances=project-id:europe-west4:db-instance-2=tcp:5432
2020/11/24 18:09:54 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/11/24 18:09:56 Listening on 127.0.0.1:5432 for project-id:europe-west4:db-instance-2
2020/11/24 18:09:56 Ready for new connections
What am I doing wrong? How do I troubleshoot or debug further?
I was able to resolve the problem by creating a service account with a different name. Just the name has changed, nothing else. If I delete the [email protected] and then use the name again, the problem still persists. I was not able to find any other reference to said account. The problem was not encountered again after my comment on Nov 30 '20.