Java application unable to find ADC when Workload Identity is enabled on GKE cluster

Question

I'm trying to migrate a service running on GKE to use Workload Identity to authenticate itself to access GCP resources. Until now, the service used SA JSON key's pointed to by the env variable GOOGLE_APPLICATION_CREDENTIALS.

The application throws this error when trying to run LOG.info("Default credentials: " + ServiceAccountCredentials.getApplicationDefault());

"Your default credentials were not found. To set up Application Default Credentials for your environment, see https://cloud.google.com/docs/authentication/external/set-up-adc.","message":"Your default credentials were not found. To set up Application Default Credentials for your environment, see https://cloud.google.com/docs/authentication/external/set-up-adc.","name":"java.io.IOException","extendedStackTrace":[{"class":"com.google.auth.oauth2.DefaultCredentialsProvider","method":"getDefaultCredentials","file":"DefaultCredentialsProvider.java","line":127,"exact":false,"location":"com.google.auth.google-auth-library-oauth2-http-1.22.0.jar","version":"1.22.0"},{"class":"com.google.auth.oauth2.GoogleCredentials","method":"getApplicationDefault","file":"GoogleCredentials.java","line":152,"exact":false,"location":"com.google.auth.google-auth-library-oauth2-http-1.22.0.jar","version":"1.22.0"},

I'm encountering the same error when the application tries to make a cloud sql connection using HikariCP.

I have followed the steps as per this Google Doc and setup a KSA and GSA, bound and annotated them as described. The GSA is assigned the role of an Editor.

On the container running my service I am able to see the corresponding GSA as the active account when I run gcloud auth list. I'm also able to the run the following curl commands successfully on the container.

curl http://metadata.google.internal/computeMetadata/v1/ -H "Metadata-Flavor: Google"

curl http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token -H "Metadata-Flavor: Google"

And I've double checked that /var/run/secrets/kubernetes.io/serviceaccount/token exists on the pod. I am also able to run commands like gcloud storage buckets list, gcloud sql instances list, gcloud alpha bq datasets list on the pod successfully.

Here's what my deployment.yaml looks like

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-app
  name: my-app
  namespace: dataservice
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: my-app
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        timestamp: "2025-01-01T00:00:00Z"
      labels:
        app: my-app
    spec:
      serviceAccountName: service-account-experimental
      nodeSelector:
        iam.gke.io/gke-metadata-server-enabled: "true"
      initContainers:
        - name: init
          image: gcr.io/google.com/cloudsdktool/cloud-sdk:alpine
          command:
            - sh
            - -c
            - |
              /bin/bash <<'EOF'
                  #!/bin/bash
                  echo starting install neo4j user;
                  neou=$(gcloud secrets versions access "latest" --secret="username-ro")
                  neop=$(gcloud secrets versions access "latest" --secret="password-ro")
                  mkdir -p /etc/my-app;
                  cp /keys/sase/neo4j_config.properties /etc/my-app/neo4j_config.properties;
                  cp /keys/sase/config.json /etc/my-app/config.json;
                  sed -i  "s/SECRETS_NEO4J_RO_USER/$neou/"  /etc/my-app/neo4j_config.properties;
                  sed -i  "s/SECRETS_NEO4J_RO_PASSWD/$neop/"  /etc/my-app/neo4j_config.properties;
                  echo done;
              EOF
          volumeMounts:
            - mountPath: /keys/sase/
              name: my-app
              readOnly: true
            - name: key-storage
              mountPath: /etc/my-app/
      containers:
      - image: my-image:latest
        name: dataservice
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 10
          failureThreshold: 2
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "2Gi"
        env:
          - name: JAVA_OPTS
            value: "-Xmx512m -Xms512m"
        ports:
        - containerPort: 8080
        volumeMounts:
          - mountPath: /etc/secrets
            name: secrets
            readOnly: true
          - mountPath: /etc/config
            name: config
            readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
        - name: secrets
          secret:
            secretName: my-secrets
        - name: config
          configMap:
            name: my-config
        - name: app-config
          secret:
            secretName: app-config
        - name: key-storage
          emptyDir: {}

Here's the serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: service-account-experimental
  namespace: dataservice
  annotations:
    iam.gke.io/gcp-service-account: gservice-account-test@my_project_id.iam.gserviceaccount.com

Answer 1

On further inspection, I had an environment variable NO_GCE_CHECK set to true in my Dockerfile which I overlooked. Setting this variable prevents the service from connecting to the metadata server.