GKE Workload Identity PermissionDenied

Question

I am trying to use Google's preferred "Workload Identity" method to enable my GKE app to securely access secrets from Google Secrets.

I've completed the setup and even checked all steps in the Troubleshooting section (https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity?hl=sr-ba#troubleshooting) but I'm still getting the following error in my logs:

Unhandled exception. Grpc.Core.RpcException: Status(StatusCode=PermissionDenied, Detail="Permission 'secretmanager.secrets.list' denied for resource 'projects/my-project' (or it may not exist).")

I figured the problem was due to the node pool not using the correct service account, so I recreated it, this time specifying the correct service account.

The service account has the following roles added:

• Cloud Build Service
• Account Kubernetes Engine Developer
• Container Registry Service Agent
• Secret Manager Secret Accessor
• Secret Manager Viewer

The relevant source code for the package I am using to authenticate is as follows:

var data = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);

var request = new ListSecretsRequest
{
    ParentAsProjectName = ProjectName.FromProject(projectName),
};

var secrets = secretManagerServiceClient.ListSecrets(request);
foreach(var secret in secrets)
{
    var value = secretManagerServiceClient.AccessSecretVersion($"{secret.Name}/versions/latest");
    string secretVal = this.manager.Load(value.Payload);
    string configKey = this.manager.GetKey(secret.SecretName);
    data.Add(configKey, secretVal);
}
Data = data;

Ref. https://github.com/jsukhabut/googledotnet

Am I missing a step in the process?

Any idea why Google is still saying "Permission 'secretmanager.secrets.list' denied for resource 'projects/my-project' (or it may not exist)?"

Answer 1

Like @sethvargo mentioned in the comments, you need to map the service account to your pod because Workload Identity doesn’t use the underlying node identity and instead maps a Kubernetes service account to a GCP service account. Everything happens at the per-pod level in Workload identity.

Assign a Kubernetes service account to the application and configure it to act as a Google service account.

1.Create a GCP service account with the required permissions.

2.Create a Kubernetes service account.

3.Assign the Kubernetes service account permission to impersonate the GCP service account.

4.Run your workload as the Kubernetes service account.

Hope you are using project ID instead of project name in the project or secret.

You cannot update the service account of an already created pod.

Refer the link to add service account to the pods.