GCP Firestore: Server request fails with Missing or insufficient permissions from GKE

Question

I am trying to connect to Firestore from code running on GKE Container. Simple REST GET api is working fine, but when I access the Firestore from read/write, I am getting Missing or insufficient permissions.

An unhandled exception was thrown by the application.
Info
2021-06-06 21:21:20.283 EDT
      Grpc.Core.RpcException: Status(StatusCode="PermissionDenied", Detail="Missing or insufficient permissions.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1623028880.278990566","description":"Error received from peer ipv4:172.217.193.95:443","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Missing or insufficient permissions.","grpc_status":7}")
         at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()

Update I am trying to provide secret to pod with service account credentails. Here is the k8 file which deploys a pod to cluster with no issues when no secrets are provided and I can do Get Operations which don't hit Firestore, and they work fine.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: foo-worldmanagement-production
spec:
  replicas: 1
  selector:
    matchLabels:
       app: foo
       role: worldmanagement
       env: production
  template:
    metadata:
      name: worldmanagement
      labels:
        app: foo
        role: worldmanagement
        env: production
    spec:
      containers:
      - name: worldmanagement
        image: gcr.io/foodev/foo/master/worldmanagement.21
        resources:
          limits:
            memory: "500Mi"
            cpu: "300m"
        imagePullworld: Always
        readinessProbe:
          httpGet:
            path: /api/worldManagement/policies
            port: 80
        ports:
        - name: worldmgmt
          containerPort: 80

Now, if I try to mount secret, the pod never gets created fully, and it eventually fails

kind: Deployment
apiVersion: apps/v1
metadata:
  name: foo-worldmanagement-production
spec:
  replicas: 1
  selector:
    matchLabels:
       app: foo
       role: worldmanagement
       env: production
  template:
    metadata:
      name: worldmanagement
      labels:
        app: foo
        role: worldmanagement
        env: production
    spec:
      volumes:
      - name: google-cloud-key
        secret:
          secretName: firestore-key   
      containers:
      - name: worldmanagement
        image: gcr.io/foodev/foo/master/worldmanagement.21
        volumeMounts:
        - name: google-cloud-key
          mountPath: /var/
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/key.json        
        resources:
          limits:
            memory: "500Mi"
            cpu: "300m"
        imagePullworld: Always
        readinessProbe:
          httpGet:
            path: /api/worldManagement/earth
            port: 80
        ports:
        - name: worldmgmt
          containerPort: 80

I tried to deploy the sample application and it works fine.

If I keep only the following the yaml file, the container gets deployed properly

- name: google-cloud-key
            secret:
              secretName: firestore-key 

But once I add the following to yaml, it fails

    volumeMounts:
    - name: google-cloud-key
      mountPath: /var/
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /var/key.json      

And I can see in GCP events that the container is not able to find the google-cloud-key. Any idea how to troubleshoot this issue, i.e why I am not able to mount the secrets, I can bash into the pod if needed.

I am using multi stage docker file made of

From mcr.microsoft.com/dotnet/sdk:5.0 AS build 
FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS runtime

Thanks

Answer 1

Looks like they key itself might not be correctly visible to the pod. I would start by getting into the pod with kubectl exec --stdin --tty <podname> -- /bin/bash and ensuring that the /var/key.json (per your config) is accessible and has the correct credentials.

The following would be a good way to mount the secret:

volumeMounts:
- name: google-cloud-key
  mountPath: /var/run/secret/cloud.google.com
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
  value: /var/run/secret/cloud.google.com/key.json

The above assumes your secret was created with a command like:

kubectl --namespace <namespace> create secret generic firestore-key --from-file key.json

Also it is important to check your Workload Identity setup. The Workload Identity | Kubernetes Engine Documentation has a good section on this.