Google cloud storage return 401 for a spring application deployed on GKE after setting GOOGLE_APPLICATION_CREDENTIALS

Question

I have a spring app that pushes data in Google Cloud Storage. I generated a new service account json file and using it during test on my windows. Evrything works fine.

Then I dockerise my app and try to deploy on Kubernetes. In order to connect the dockerise app with the cloud storage I created a secret with my service account json file

kubectl create secret generic cloud-storage-credentials \
 --from-file=cloudstorage.json=cloud-storage-credentials.json

Then I mount in my deployment file the cloud storage file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "{{ include "m-ebook.name" . }}-deployment"
  labels:
    app: {{ include "m-ebook.name" . }}
    tier: backend
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ include "m-ebook.name" . }}
      tier: backend
  template:
    metadata:
      labels:
        app: {{ include "m-ebook.name" . }}
        tier: backend
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            #Cloud storage
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /var/secrets/google/cloudstorage.json
            #Cloud sql
            - name: DB_HOST
              value: 127.0.0.1:3306
            # These secrets are required to start the pod.
            # [START cloudsql_secrets]
            # The db name is set directly in the back end propeties files
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: password
        #Rabbitmq cookie erlang (needed to connect to rabbitmq)
            - name: RABBITMQ_ERLANG_COOKIE
              valueFrom:
                secretKeyRef:
                  name: rabbitmq
                  key: erlangCookie
          # [END cloudsql_secrets]
        # Change <INSTANCE_CONNECTION_NAME> here to include your GCP
        # project, the region of your Cloud SQL instance and the name
        # of your Cloud SQL instance. The format is
        # $PROJECT:$REGION:$INSTANCE
        # [START proxy_container]
        - name: cloudsql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.11
          command: ["/cloud_sql_proxy",
                  "-instances=gara-261618:europe-west1:gara-postgresql-server=tcp:3306",
                  "-credential_file=/secrets/cloudsql/credentials.json"]
          # [START cloudsql_security_context]
          securityContext:
            runAsUser: 2  # non-root user
            allowPrivilegeEscalation: false
          # [END cloudsql_security_context]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: cloud-storage-credentials-volume
              mountPath: /var/secrets/google
              readOnly: true
        # [END proxy_container]
          ports:
            - name: http
              containerPort: {{ .Values.service.internalPort }}
              protocol: TCP
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      imagePullSecrets: 
        - name: registry-gitlab-secrets
      # [START volumes]
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: cloud-storage-credentials-volume
          secret: #The name of the secret as defined in create secret generic cloud-storage-credentials
            secretName: cloud-storage-credentials
      # [END volumes]

EDIT The secret is a file generated from google and looks like :

{
  "type": "service_account",
  "project_id": "myproject-261618",
  "private_key_id": "3d1625a7428367cfb274251",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgk...Z7XSQik\nYWSPGLxNDlopi+DLDzzHvJtO\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "1023295593410734556",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/name%40nameofmyproject.iam.gserviceaccount.com"
}

EDIT

kubectl get secret cloud-storage-credentials -o yaml
apiVersion: v1
data:
  cloudstorage.json: ewogICJ0eXBlIjogInNlcnZpY2V...DUwOV9jZXJ0X3VybCI6ICJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9yb2JvdC92MS9tZXRhZGF0YS94NTA5L2dhcmEtY2xvdWQtc3RvcmFnZS1pbnQtZGV2JTQwZ2FyYS0yNjE2MTguaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iCn0K
kind: Secret
metadata:
  creationTimestamp: "2020-02-27T16:19:27Z"
  name: cloud-storage-credentials
  namespace: default
  resourceVersion: "23609"
  selfLink: /api/v1/namespaces/default/secrets/cloud-storage-credentials
  uid: ecb4821d-597c-...010af001ab
type: Opaque

EDIT When I try to read the property inside my code and check if the file exists, the property is read, but the doesn't exists.

5    [http-nio-9103-exec-3] INFO  com.gara.mebooks.services.googlecloud.implement.GcloudStorageServiceImpl  - /var/secrets/google/cloudstorage.json
5    [http-nio-9103-exec-3] INFO  com.gara.mebooks.services.googlecloud.implement.GcloudStorageServiceImpl  - is the file exists ? false

I don't why but when I call my services, I got a 401 from Google. Please what did I do wrong ? I enter inside the container and when I type echo $GOOGLE_APPLICATION_CREDENTIALS I got /var/secrets/google/cloudstorage.json but when I try to cat /var/secrets/google/cloudstorage.json, I got file not found exception. Is it normal ?

Thanks in advance

Answer 1

Your deployment has two different containers on it. You are setting your env var in the first container and mounting the volume in the second.

Here's how your code should look like.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "{{ include "m-ebook.name" . }}-deployment"
  labels:
    app: {{ include "m-ebook.name" . }}
    tier: backend
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ include "m-ebook.name" . }}
      tier: backend
  template:
    metadata:
      labels:
        app: {{ include "m-ebook.name" . }}
        tier: backend
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            #Cloud storage
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /var/secrets/google/cloudstorage.json
            #Cloud sql
            - name: DB_HOST
              value: 127.0.0.1:3306
            # These secrets are required to start the pod.
            # [START cloudsql_secrets]
            # The db name is set directly in the back end propeties files
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: password
          volumeMounts:
            - name: cloud-storage-credentials-volume
              mountPath: /var/secrets/google
              readOnly: true
        #Rabbitmq cookie erlang (needed to connect to rabbitmq)
            - name: RABBITMQ_ERLANG_COOKIE
              valueFrom:
                secretKeyRef:
                  name: rabbitmq
                  key: erlangCookie
          # [END cloudsql_secrets]
        # Change <INSTANCE_CONNECTION_NAME> here to include your GCP
        # project, the region of your Cloud SQL instance and the name
        # of your Cloud SQL instance. The format is
        # $PROJECT:$REGION:$INSTANCE
        # [START proxy_container]
        - name: cloudsql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.11
          command: ["/cloud_sql_proxy",
                  "-instances=gara-261618:europe-west1:gara-postgresql-server=tcp:3306",
                  "-credential_file=/secrets/cloudsql/credentials.json"]
          # [START cloudsql_security_context]
          securityContext:
            runAsUser: 2  # non-root user
            allowPrivilegeEscalation: false
          # [END cloudsql_security_context]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
        # [END proxy_container]
          ports:
            - name: http
              containerPort: {{ .Values.service.internalPort }}
              protocol: TCP
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      imagePullSecrets: 
        - name: registry-gitlab-secrets
      # [START volumes]
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: cloud-storage-credentials-volume
          secret: #The name of the secret as defined in create secret generic cloud-storage-credentials
            secretName: cloud-storage-credentials
      # [END volumes]